views:

189

answers:

5

I have a lot of python unit tests for a project and it's getting to the point where it takes a long time to run them. I don't want to add more because I know they're going to make things slower. How do people solve this problem? Is there any easy way to distribute the test execution over a cluster?

+2  A: 

Profile them to see what really is slow. You may be able to solve this problem with out distribution. If the tests are truly unit tests then I see not many problems with running the tests across multiple execution engines.

Preet Sangha
+1  A: 

The first part of the solution to this problem is to only run the tests one needs to run. Between commits to a shared branch, one runs only those tests that one's new work interacts with; that should take all of five seconds. If one adopts this model, it becomes vital to make a point of running the entire test suite before committing to a shared resource.

The problem of running the full test suite for regression purposes remains, of course, though it's already partially addressed by simply running the full suite less often. To avoid having to wait around while that job runs, one can offload the task of testing to another machine. That quickly turns into a task for a continuous integration system; buildbot seem fairly appropriate to your use case.

You should also be able to distribute tests across hosts using buildbot, firing off two jobs with different entry points to the test suite. But I am not convinced that this will gain you much over the first two steps I've mentioned here; it should be reserved for cases when tests take much longer to run than the interval between commits to shared resources.

D'A

[Caveat lector: My understanding of buildbot is largely theoretical at this point, and it is probably harder than it looks.]

darch
This seems like very bad advice, in the context of unit tests specifically. After editing the code, *all* the unit tests should be run again before committing.Your advice applies well to tests larger in scope than unit tests (e.g. integration tests, regression tests, performance tests, etc.) but not to unit tests.
bignose
It's worth noting here that my model of revision control favors tightly atomic commits, the majority of them not exposed to the team at large. This makes running unit tests across the entire application for every commit too burdensome for too little payoff.Thus my emphasis on re-running the entire unit test suite before committing to a *shared* space, to catch the things you broke while you were noodling around on your own.
darch
+1  A: 

While coding, only run the tests of the class that You have just changed, not all the tests in the whole project.

Still, it is a good practice to run all tests before You commit Your code (but the Continuous Integration server can do it for You).

Reef
+2  A: 

See py.test, which has the ability to pass unit tests off to a group of machines, or Nose, which (as of trunk, not the currently released version) supports running tests in parallel with the multiprocessing module.

jnoller
Very interesting to learn about distributing tests with py.test. I'm writing a loadtesting framework to stress-test an n-tier application by starting lots of "virtual" clients accross many different hosts. I have to write code to co-oridinate this and the code in py.test will certainly come in handy as a reference.
jkp
+3  A: 

You can't frequently run all your tests, because they're too slow. This is an inevitable consequence of your project getting bigger, and won't go away. Sure, you may be able to run the tests in parallel and get a nice speedup, but the problem will just come back later, and it'll never be as it was when your project was small.

For productivity, you need to be able to code, and run relevant unit tests and get results within a few seconds. If you have a hierarchy of tests, you can do this effectively: run the tests for the module you're working on frequently, the tests for the component you're working on occasionally, and the project-wide tests infrequently (perhaps before you're thinking of checking it in). You may have integration tests, or full system tests which you may run overnight: this strategy is an extension of that idea.

All you need to do to set this up is to organize your code and tests to support the hierarchy.

Paul Hankin