Test Distribution

Question

At my work we are running a group of tests that consist of about 3,000 separate test cases. Previously we were running this entire test suite on one machine, which took about 24-72 hours to complete the entire test run. We now have created our own system for grouping and distributing the tests among about three separate machines and the tests are prioritized so that the core tests get run first for more immediate results and the extra tests run when there is an available machine.

I am curious if anyone has found a good way to distribute their tests among several machines to reduce total test time for a complete run and what tools were used to achieve that. I've done some research and it looks like TestNG is moving in this direction, but it looks like it is still under quite a bit of development.

We don't plan on rewriting any of our tests, but as we add new tests and test new products or add-ons I'd like to be able to deal with the fact that we are working with very large numbers of tests. On the other hand, if we can find a tool that would help distribute our Junit 3.x tests even in a very basic fashion, that would be helpful since we wouldn't have to maintain our own tooling to do that.

Answer 1

I've seen some people having a play with distributed JUnit. I can't particularly vouch for how effective it is, but the other teams I've seen seemed to think it was straight forward enough. Hope that helps.

Answer 2

@GaryF, distributed JUnit looks like pretty interesting, I'll have to look at it some more but it seems like it could definitely help.

Answer 3

Our build people use Mozilla Tinderbox. It seems to have some hooks for distributed testing. I'm sorry not to know the details but I thought I would at least pass on the pointer to you.

It's also nice coz you can find out immediately when a build breaks, and what checkin might have been the culprit.

http://www.mozilla.org/tinderbox.html

Answer 4

There's also parallel-junit. Depending on how you currently execute your tests its convenience may vary - the idea is just to multithread on a single system that has multiple cores. I've played with it briefly, but it's a change from how we currently run our tests.

Hudson, the continuous integration engine I use, also has some ways to distribute test running (separate jobs aggregated results in one).