What automated build systems are you using? How long do your builds take?

Question

My current project has 5 separate automated builds that kick off after each check-in:
Unit Tests (DB calls mocked out): ~6 Minutes
Integration Tests (just to the DB): ~40 Minutes
Website 1 UI (Selenium, from UI to DB): ~80 Minutes
Website 2 UI1 (Selenium, from UI to DB): ~90 Minutes
Website 2 UI2 (Selenium, from UI to DB): ~100 Minutes

We are using Maven2, JUnit and Selenium.

One strategy that I think will reduce drastically reduce Integration test time, is moving as many Integration tests as possible into the Unit Tests, and simply using the Integration project to test persistence to the DB.

I'm wondering what strategies you've found that have helped to reduce build times on large projects.

Answer 1

We have just about the same build runtimes, maybe slightly less on selenium (I'd say around 3x50 minutes runtime, same site tests on firefox, ie and opera). Our solution was to throw more cpu at it, and we have a clustered bamboo environment of totally 7 dual-core nodes.

We found that running selenium-rc and the browser on a seperate box from the web-container/selenium test improved selenium runtimes quite significantly.

Answer 2

Team System: under 1 hour for everything

Answer 3

I assume your project consists of several subprojects. If it does not you may want to refactor the code base so that it does. This way you can chose to only build specific parts of the overall application, test the individual part then run your integration tests when you’ve worked out the bugs on the subproject you are working on.

Answer 4

I'm using GNU make to automate everything. Depending it takes from 2 minutes to 30 minutes depending on the project.

Answer 5

We use Hudson to build a large C# solution containing a handful of windows services, web services, and executables. Around 800 or so tests via MSTest.exe and it takes about 12-15 minutes per build. Tests take up a large portion since MSTest is slooow.

To cut down on time we've tried to limit the number of assemblies produced to run through MSTest since the setup and teardown seems to take a while.

Edit: Our build also deploys to our cert env which includes web services, windows services, executables and database deployment. Just to give you an idea of scope

Answer 6

Erlang Common Test - mostly focussing on System Tests as opposed to Unit Tests.

Answer 7

All our automated builds are done with Team City.

The most automated project we work with is effectively a compiler. Our test suite consists of around 20000 test queries that are compiled and run, which is run on every checkin. This used to take the best part of an hour, but keeping the complete test suite in a RAM drive on the build machine (rather than checking it out each time) reduces this to a few minutes.

Every night a second build is triggered that runs these same tests, but under 4 different profiles, whilst running under NCover which produces a code coverage report. This scenario takes several hours, which is why it is done as a nightly build. Other internal reports are produced at the same time to ensure that everything is working as it should.

Updates to the test suite itself are triggered separately and checked out to the RAM drive ready for the next run. Checking the tests out was otherwise taking most of the build time. Part of the test suite comes from a remote CVS repository out of our control and even querying this for updates added a few minutes to the build time, so this is also done in the "Update tests" build. This loose coupling has meant that we have had to restrict builds of this project to one machine, but since feedback is so quick, this is not a big problem.

It has proved immensely helpful to keep the "regular" test build as fast as possible whilst retaining high coverage of the codebase. Any slow tests (more than a second or so) have been pulled through to the nightly build. In our case keeping the tests in a RAM drive has really helped, although our scenario is rather specialist. I guess that mocking your database is the nearest equivalent. My advice is to keep one build "lean and mean" by removing any "chunky" or slow tests, allowing a snappy response to know you probably haven't broken anything. The other builds can be run on separate machines, or at night in order to keep the response quick from the fast build.

For perspective the longest automated build we have (for another project) sometimes takes more than a day, although thankfully it is not one that needs to be run regularly.

Answer 8

At my current client, unfortunately, BuildForge (from ClearCase SCC): anywhere from one to three days. Seriosuly. I don't really know what they do in a build or why it takes so long.