NUnit Test Run Order

Question

Hi,

By default nunit tests run alphabetically. Does anyone know of any way to set the execution order? Does an attribute exist for this?

Any help would be greatly appreciated.

Thanks Zaps

Answer 1

Your unit tests should each be able to run independently and stand alone. If they satisfy this criterion then the order does not matter.

There are occasions however where you will want to run certain tests first. A typical example is in a Continuous Integration situation where some tests are longer running than others. We use the category attribute so that we can run the tests which use mocking ahead of the tests which use the database.

i.e. put this at the start of your quick tests

[Category("QuickTests")]

Where you have tests which are dependant on certain environmental conditions, consider the TestFixtureSetUp and TestFixtureTearDown attributes, which allow you to mark methods to be executed before and after your tests.

Answer 2

Why would you want to do this? It sounds like you have a dependency on the run order, which is a bad thing. You need to reconsider why you want this. Unit tests should run in isolation and be totally independent of others.

It sounds like you are creating candidates for the test smell Erratic Tests.

Answer 3

You should not depend on the order in which the test framework picks tests for execution. Tests should be isolated and independent. In that they should not depend on some other test setting the stage for them or cleaning up after them. They should also produce the same result irrespective of the order of the execution of tests (for a given snapshot of the SUT)

I did a bit of googling. As usual, some people have resorted to sneaky tricks (instead of solving the underlying testability/design issue

• naming the tests in an alphabetically ordered manner such that tests appear in the order they 'need' to be executed. However NUnit may choose to change this behavior with a later release and then your tests would be hosed. Better check in the current NUnit binaries to Source Control.
• VS (IMHO encouraging the wrong behavior with their 'agile tools') has something called as "Ordered tests" in their MS testing framework. I didn't waste any time reading up but it seems to be targeted towards the same audience

See Also: characteristics of a good test

Answer 4

This looks like a duplicate but you can see my answer to that here

Answer 5

Wanting the tests to run in a specific order does not mean that the tests are dependent on each other - I'm working on a TDD project at the moment, and being a good TDDer I've mocked/stubbed everything, but it would make it more readable if I could specify the order which the tests results are displayed - thematically instead of alphabetically. So far the only thing I can think of is to prepend a_ b_ c_ to classes to classes, namespaces and methods. (Not nice) I think a [TestOrderAttribute] attribute would be nice - not stricly followed by the framework, but a hint so we can achieve this

Answer 6

For all you advocating that order is irrelevant and that each test should be executable by itself, I fully agree, but consider the following:

• I have a method A and a method B.
• Method B relies on method A.
• I have a unit test TestA that tests method A,
• and TestB that tests method B (of
  course I wrote the tests first!).

So if there is an error in B, then TestB fails. However, if A has an error, TestB as well as TestA fails. And yes, we can figure out in TestB that it was method A that failed, but would it not be nice to have the first, smallest, point of failure detected first?

So if there was an execution order for the tests TestA would run before TestB, and both would fail, but TestA's failure would be at the top of the list, so I will investigate it first and instantly find the problem is with method A.

Now take this into the real world, I have 100 or 1000 unit tests, that test inter-related parts of the SUT. All tests are still independent and can be executed standalone, but if I run all of them, and they run in order, I immediately find the first point of failure.

And that's a good reason for test order. Not to setup a dependency chain, but to set up targeted failure detection.

Answer 7

I am also in need of a way to run my tests in order, but for a different reason. I am a QA engineer, and am implementing selenium remote control integrated with Cruisecontrol. My implementation of RC uses C#, Visual Studio, .net 2.0, and uses Nunit to drive the tests. (yes, this is how it's supposed to be done). So, from a QA perspecitve, I have a variety of tests i want to run, which are order dependant. For example, consider a basic CRUD, you need to create the item before you can update or delete it, but if the update fails you still want to test the delete.

Theoretically I could add create and delete to the setup and teardown, but that ends up adding a lot of overhead, as I will have many many tests that will be run, and creating/deleting for each test will seriously prolong the execution time.

I have two options, I can have 1 unit test, which calls seperate functions for create, read, update, delete. However, if one function fails, the remainder will not even run. In addition, the reporting back to the cruisecontrol server is not very clear. Alternately, I could create 4 seperate unit tests, but I need them to be run in order. This solves both problems, all tests will be run each time, even if one fails, and the reporting is much easier to read, and determine what functionality is actually broken.

I'm going to have to just alphabetize my tests, but there has to be a better solution to this.