How do you structure your NUnit tests on a large project?

Question

I am interested in seeing if I can improve the way we use NUnit in a Visual Studio solution containing 30+ projects.

First, would you have one assembly of tests for every assembly in the solution, or would you try to keep the number of test assemblies down? I started off creating many test assemblies, but I think this is costing us a lot in terms of build time.

Second, what strategy do you use for managing those tests that are long-running, or require special environment configuration? I would like to write an MSBuild script that automates the running of our unit tests, but it needs to skip over the tests that would take too long or would not work on the build machine.

Answer 1

To answer your first question:

You can use categories to keep everything organised in a single assembly if that's how you want to go. As to whether it would cut down on build time, I would hazard a guess at perhaps, I don't think it'd be a huge amount overall though.

To answer your second:

You can use categories and the [Explicit] and [Ignore] attribute to tell NUnit which tests to run, and which tests you have to tell it to run before it will. You might also want to look at [Platform] or some of the other attributes, depending on what exactly your requirements on 'environment' are.

For instance, you could add an Explicit tag to all your long running tests. Then you would have to run these tests explicitly, NUnit won't run them automatically. Or add all these tests to a category, then when you run NUnit, tell it to explicitly not run that category.

Answer 2

For your case I would put it down to a few options:

• Consolidate your existing projects to fewer ones, consequently consolidating corresponding test projects for them - Try to think if you really need 30 projects (e.g., 30 assemblies) or if it is better to simply combine them to ease the number of dependencies and build time.

• Put your tests in their corresponding project - While this may elicit a violent reaction, think about it: considering the huge number of tests involved the overhead of getting a slightly larger assembly might be worth the reduced overhead of managing twice the number of projects you actually need.

• Combine all the unit tests into a few projects - If you're adamant about keeping each assembly clean and separate, you can just opt to combine all the unit tests, probably in terms of relevance with each other.

Consequently you could also make solution configurations that do not include unit tests - if build time is so much of a concern to you. This is especially useful if you're just running tests on demand. If you're using continuous integration, it shouldn't be a problem: you can have multiple CI configurations, some which include building tests, some that do not.

Answer 3

This a great question and raises some concerns that I have about managing large set of unit tests for long running, large projects.

My strategy is to have one unit test project for each project in your solution that is business or framework related. Unit tests are poorly suited to testing the UI aspects of a project and I generally leave this to scipted testing. The project is built using a continous integration (CI) server that is fired every time a developer commits a chunk of work. The CI server also runs all of the unit test as well generating unit test coverage statistics. The CI environment uses MSBuild to build the solution.

Over time it is worth assessing whether all of your unit tests are necessary. A lot of the regression testing effort will shift to scripted testing and the overhead of maintaining unit tests can be costly. Ideally I would like all unit tests to be maintained over the life of the project but the maintenance overhead can sometimes be prohibitive. As a rule of thumb unit tests are maintained for business rules and the solution framework but not for UI, reporting or workflow. Obviously this will depend on the particulars of your project.

Answer 4

Just as an aside, we find TestDriven.Net to be an excellent VisualStudio add-in front end for NUnit, free for open source use / students.