Best way to automatically check out and compile Eclipse projects with Ant in Hudson or another CI tool?

Question

We have several products which have a lot of shared code and which must be maintained several versions back.

To handle this we use a lot of Eclipse projects, some contain library jars, and some contain shared source code (in several projects to avoid getting a giant heap with numerous dependencies while being able to compile everything from scratch to ensure that source and binaries are consistent). We manage those with projectSet.psf's as these can directly pull all projects out from CVS and leave a fully prepared workspace. We do not do ant builds directly or use maven.

We now want to be able to put all these projects and their various versions in a Continous Integration tool - I like Hudson but this is just a matter of taste - which essentially means that we need to get an automatic way to check out the projects to a fresh workspace, and compile the source folders as described in the project-files in each project. Hudson does not provide such an approach to build a project, so I have been considering what the best way to approach this would be.

Ideas have been

• Find or write an ant plugin/converter that understands projectSet.psf's and map to cvs-checkout and compile tags.
• Create the build.xml files from within Eclipse and use those. I tried this, and found the result to be verbose and with absolute locations which is not good with automatic tools putting files where they want to.
• Write a Hudson plugin which understands projectSet.psf's to derive a configuration and build it.
• Just bite the bullet and manually create and update the CI configuration whenever stuff breaks - I don't like this :)

I'd really like to hear about other peoples experiences so I can decide how to approach this.

---

Edit: Another option might be using a CI which knows better about Eclipse projects and/or project sets. We are not religious - this is just a matter of getting stuff running without having to do everything ourselves. Would Cruise Control be a better option perhaps? Others?

---

Edit: Found that ant4eclipse has a "Team Project Set" facility. http://ant4eclipse.sourceforge.net/

---

Edit: Used the ant4eclipse and ant-contrib ant extensions to build a complete workspace as a sjgned runnable jar file similar to the Runnable Jar facility in Eclipse 3.5M6. I am still depending on Eclipse to create the initial empty workspace, and extract the ProjectSet, so that is the next hurdle.

---

Edit: Ended up with a dual configuration, namely that Hudson extracts the same set of modules as listed in the ProjectSet.pdf file from CVS (which needs to have the same tag) causing them to be located next to each other. Then ant4eclipse works well with the projectSet.psf file embedded in the main module. Caveat: Module list in Hudson must be manually updated, and it appears that a manual workspace cleanup is needed afterwards to let Hudson "discover" that there is more projects now than earlier. This has now worked well for us for a couple of months, but it was quite tedious to get everything working inside the ant file.

---

Edit: The "Use Team Projects" with ant4eclipse and a Ctrl-A, Ctrl-C in Project Panel with a Ctrl-V in the CVS projects in Hudson has turned out to work well enough for us to live with (for mature projects this is very rarely changed). I am awaiting the release of ant4eclipse 1.0 - http://www.ant4eclipse.org/, currently in milestone 2 - to see how much homegrown functionality can be replaced with ant4eclipse things.

Edit: ant4eclipse is as of 20100609 in M4 so the schedule at http://www.ant4eclipse.org/node?page=1 is slipping somewhat.

Answer 1

I'm not completely sure I understand the problem, but it sounds like the root issue is that you have many projects, some of which are dependent on others. Some of the projects that are closer to the "leaf" of the dependency tree need to be able to use "stable" (or previously "released") versions of the more "core" projects.

I solve exactly this problem using Hudson, ant, and ivy. I follow a pattern demonstrated by Clark in Pragmatic Project Automation (he doesn't demonstrate the dependency problems and solutions, and he uses CruiseControl rather than hudson.)

I have a hand-written ant build file (we call it "cc-build.xml", because of our CruiseControl roots.) This file is responsible for refreshing the working space for the project from the CM repository and labeling the contents for future reference. It then hands off control to another hand-written ant build file (build.xml) that is provided by each project's developers. This project is responsible for the traditional build steps (compile, packaging, etc.) It is required to spit out the installable artifacts, unit test reports, etc, to the Hudson artifacts directory. It is my experience that automatically generated build files (by Eclipse or other similar IDE's) will never get close to getting this sufficiently robust for use in a CI scenario.

Additionally, it uses ivy to resolve its own dependencies. Ivy supports precisely-specified dependency versions (e.g. "use version 1.1") and it supports "fuzzy versions" (e.g. "use version 1.1+" or "use the latest version in integration status.") Our projects typically start out specifying a very "fuzzy" version for internal projects under ongoing development, and as they get close to a release point, they "freeze" the dependency version so that stuff stops moving underneath them.

The non-leaf projects (projects that are dependents for other projects) also use ivy to publish their artifacts to our internal ivy repository. That repository keeps all past builds of the dependents, so that any project can always depend on any other previous version.

Lastly, each project in Hudson is configured to have a build trigger that causes a rebuild when any of its dependent projects successfully build. This causes them to get built again with the (possibly) new ivy dependent version.

It is worth noting that once you get this up and running, consistent automated "labeling" or "tagging" of an automated build's inputs is going to be critical for you - otherwise troubleshooting post-build problems is going to result in having to untangle a hornet's nest to find the original source.

Getting all of this setup for our environment took quite a bit of effort (primarily in setting up the ivy repository and ant build files,) but it has paid for itself many times over in saved headaches in manually managing the dependencies and decreased troubleshooting effort.

Answer 2

Write a Hudson plugin which understands projectSet.psf's to derive a configuration and build it.

That seems like the winning answer to me.

I work with CruiseControl rather than Hudson but in my experience if you can create a plugin that solves your problem it will quickly payoff. And it is generally pretty easy to write a plugin that is custom fit for your solution as opposed to one that needs to work for everyone in a similar situation.

Answer 3

I have tried both Cruise Control (CC) and Hudson for our CI solution. We (as a company) decided on Hudson. But for your question "Does CC support Eclipse project build" the answer is no as far as I know. CC supports many more different build tools and Source Control systems but it is a bit more difficult to configure and use. As for Hudson, it is more simple to configure and use it. We developed our custom plugins for both CC and Hudson for the parts of our build cycle that they do not provide as is. As for plugin development, if you know / use Maven, Hudson is simpler too. But if you are not familiar to Maven, first you need to learn the basic usage of maven to successfully develop a Hudson plugin. But once you understand the basic usage of maven, plugin development, test and even debug is simpler in Hudson.

For your specific problem, I can think of a solution that makes use of Eclipse plugins as well. You can develop your own Eclipse plugin that for instance gets the psf files from a (configurable) folder, and use Eclipse internals to process these psf's. I mean you can use existing Eclipse source codes that takes a psf file, check-outs it's project definitions and compile these projects. This Eclipse plugin of yours may have a preference page (which you can access by Eclipse -> Window -> Preferences) and configure which folder it will use to look for psf files. Your Eclipse plugin should also have a way to start psf processing without user interaction. For this, you can use ipc to trigger your process. I mean your Eclipse plugin can listen for a port, and you can write another java application that will connect to your plugin through this port and trigger its process. As for CI part, you can use either CC or Hudson and use their external process execution support. If you are using Windows, you can write a bat file (for Linux sh file) that first launchs Eclipse that has your plugin installed. Then it launches your java application that will communicate with your Eclipse plugin to trigger your process. From your CI tool you will need to run your bat / sh file to trigger your process.

Answer 4

Note: for now I have handcraftet an ant file using ant4eclipse to generate "executable jar files" which has been very tedious, but the long term goal is to teach Hudson about eclipse projects.

It seems I am in the same boat. You mentioned you created your own plugin for this. Are you still using it or did you find anything that worked for you?