I've got a large acceptance test (~10 seconds per test) test suite written using NUnit. I would like to make use of the fact that my machines are all multiple core boxes. Ideally, I'd be able to have one test running per core, independently of other tests.
There is PNUnit, but it's designed for testing for threading synchronization issues and things like that, and I didn't see an obvious way to accomplish this.
Is there a switch/tool/option I can use to run the tests in parallel?
It would be a bit of a hack, but you could split the unit tests into a number of categories. Then, start up a new instance of NUnit for each category.
Edit: It looks like they have added a /process option to the console app. The command-line help states this is the "Process model for tests: Single, Separate, Multiple". The test runner also appears to have this feature.
Edit 2: Unfortunately, although it does create separate processes for each assembly, the process isolation option (/process from the command line) runs the agents one at a time.
In this article it is mentioned that in order to speed up tests the poster runs multiple instances of NUnit with command parameters specifying which tests each instance should run.
FTA:
I ran into an odd problem.
We use nunit-console to run test on our continuous integration server. Recently we were moving from Nunit 2.4.8 to 2.5.5 and from .Net 3.5 to 4.0. To speed up test execution we run multiple instances of Nunit in parallel with different command line arguments
- We have two copies of our test assemblies and the nunit binaries in folder A and B.
- In folder A we execute
nunit-console-x86.exe Model.dll Test.dll /exclude:MyCategory /xml=TestResults.xml /framework=net-4.0 /noshadow
- In folder B we execute
nunit-console-x86.exe Model.dll Test.dll /include:MyCategory /xml=TestResults.xml /framework=net-4.0 /noshadow
If we execute the commands in sequence both run successfully. But if we execute them in parallel only one succeeds. As far as I can tell it's the one that first loads the test fixtures. The other fails with the message "Unable to locate fixture".
Is this problem already known? I could not find anything related in the bug list on launchpad. BTW Our server runs Windows Server 2008 64-bit. I could also reproduce the problem on Windows 7 64-bit.
Assuming this bug is fixed or you are not running the newer version(s) of the software mentioned you should be able to replicate their technique.
Update
TeamCity looks like a tool you can use to automatically run NUnit tests. They have an NUnit launcher discussed here that could be used to launch multiple NUnit instances. Here is a blog post discussing the mergind of multiple NUnit XML results into a single result file.
So theoretically you could have TeamCity automatically launch multiple NUnit tests based on however you want to split up the workload and then merge the results into a single file for post test processing.
Is that automated enough for your needs?
Just because PNUnit can do synchronization inside test code doesn't mean that you actually have to use that aspect. As far as I can see there's nothing to prevent you from just spawning a set and ignoring the rest till you need it.
BTW I don't have the time to read all of their source but was curious to check out the Barrier class and it's a very simple lock counter. It just waits till N threads enter and then sends the pulse for all of them to continue running at the same time. That's all there is to it - if you don't touch it, it won't bite you.
Might be a bit counter intuitive for a normal threaded development (locks are normally used to serialize access - 1 by 1) but it is quite a spirited diversion :-)
The standard nunit runner does not support running tests in parallel. You can create your own test runner to run the tests in parallel (using your current nunit tests). I'm not sure why the nunit team haven't done this already.
Alternatively, MBUnit has the option of creating parallelizable tests, and as MBUnit has pretty much the same syntax as NUnit, it might not take that much effort to make the switch.