Language Agnostic Build Management System

Question

Several times in my career, I have worked in a software group that determined that

a) We needed a build/test system
b) We should write our own
c) We can have a developer spend a week, get it done and they shouldn't have to touch it again

Every time, this has resulted in a system that only seems to work for the person that wrote it and requires their constant attention. I've spent time on several occasions looking for a tool that I could grab that would serve our needs, but come up empty-handed. Generally, tools like this server a very narrow market. I'm at the point again of needing something like this. Is there something out there, or do we write it again?

Here are my requirements in priority order (the last few are just nice to have):

1. Ability to handle a multi-project build. We have several components that both provide things other components use and use things from other components. A developer should be able to check out 1 component and make changes without having to build the world. Dependencies outside the project should be pulled in automatically. So some way to be able to push and pull the built objects to a server is critical for this. Another aspect of this is the ability to be able to pull down all dependencies to a local directory for development on the road.

2. Don't worry about exactly how things get built. This may sound weird, but I don't want the build system to worry about compiling my code. There are already great tools that do this for every language - Ant, CMake, etc. I just want to tell it what to call to make things build, and what output it should care about. This way, Project A can be in Java, Project B can be in C++, you get the idea.

3. Have some way to run tests on the output

4. Show the current build/test results on a web page

5. Email the results

6. Integration with RCS (we use svn)

Answer 1

Apache's Continuum and Atlassian's Bamboo will meet #2-6.

Doing #1 in a language-agnostic way is a little tougher.

Maven and Ivy both work for Java dependencies.

Answer 2

I've used Visual Build, which I found to be a very useful way to tie all these disparate compilers, test runners, and other things together into a flexible system. Think Windows Scheduler on major steriods. It's basically a big task execution engine, with ready-made tasks included for all the major commercial source control systems, compilers, and so forth. It can create folders, send emails, and a bunch of other things all with a fairly simple user interface.

Answer 3

I've researched, but never used, AntHill. There's an open source version and a commercial version. I think it will do what you want, but it would be helpful if you gave more requirements, like does it need to do automated overnight builds, what you use for source control, etc.

In my current company, I wrote my own. The way mine works is exactly what you say; it has no idea how to build the software. You give it a bunch of command lines to run (stored in a database), it captures the exit value and stdout and stderr, and if the exit value is non-zero, it marks the build as broken. We have about 8 projects in there that can build and run unit tests. All of them start out by wiping the directory and getting the source fresh out of subversion.

Answer 4

Here is a short list of tools that work with .NET and other languages: Automated Build Tools

Answer 5

I would recommend that if you are going to write a testing system that you consider using the Test Anything Protocol. TAP has been in use for 20+ years, and widely used, especially for CPAN modules.

TAP's general format is:

1..N
ok 1 Description # Directive
# Diagnostic
....
ok 47 Description
ok 48 Description
more tests....

For example, a test file's output might look like:

1..4
ok 1 - Input file opened
not ok 2 - First line of the input valid
ok 3 - Read the rest of the file
not ok 4 - Summarized correctly # TODO Not written yet

Go to testanything.org for more information.

*The examples are copied from http://en.wikipedia.org/wiki/Test_Anything_Protocol*

Answer 6

Adam,

I noticed the Anthill response, and as an Anthiller, I have to say he was right. One of the things that Anthill does real well is let you define dependencies between projects. The open source version is Java focused, while the commercial tool is language agnostic despite the name.

It would allow you to define dependencies between projects (or parts of projects) based on criteria like status (latest successful build, or latest build approved by QA or...) and/or build number or branch. It something we're pretty proud of. At build time, build artifacts are shuttled around between servers, caches are kept for performance help, and all that good stuff.

As for the other criteria:
2) Anthill (like most build automation tools) will run your existing build scripts, usually without modification.
3) Tests (again your existing technologies) can be run at build time, or against an existing build.
4) Results are shown on the web (again typical of build automation tools)
5) Email is easy
6) We have over a dozens SCM integrations, and use SVN internally so that integration is naturally one of our best.

cheers,

eric