unit testing failing when run in batch

Question

I am new to unit testing. I have created various tests and when I run test each one by one, all tests passing. However, when I run run on the whole as a batch, some tests failing. Why is that so? How can I correct that?

Answer 1

Sometimes final conditions of one test have an impact on initial conditions of the next.

Manual run and batch run may have different behaviour regarding how initial conditions of each test are set.

Answer 2

Probably some of your tests are dependent on the prior state of the machine. Unit tests should not depend on the previous state of the process/machine, so you should look at the failing tests and work out what they are depending on.

Answer 3

To resolve this issue it is important to follow certain rules when writing unit tests. Some of the rules are easy to follow and apply while other may need further considerations depending your circumstances.

Each test should set up a unique set of data. This is in particular important when you work with persistent data, e.g. in a database. When the test creates a user with a particular user id, then write the test so that it uses a different user id each time. For example (C#):

var user = new User(Guid.NewGuid());

At the end of each test, cleanup the data that the test has created. For example in the tear down method remove the data that you created (C#, NUnit):

[TearDown]
public void TheTearDownMethod() {
    _repository.Delete(_user);
}

Variations are possible, e.g. when you test against a database you may choose to load a backup just before you run the test suite. If you craft your tests carefully you don't need to cleanup the database after each test (or subset of tests).

To get from where you are now (each test passes when run in isolation) to where you would like to be start with running the first two tests in sequence, make them pass. Then run three in sequence, make them pass, etc. In each iteration identify what previous test causes the added test to fail. Resolve that dependency. This way you learn a lot about your tests but also how to avoid writing tests that depend on each other.

Once the suite passes in one patch run it frequently so that you detect dependency as early as possible.

This doesn't cover all scenarios and variations but hopefully gives you a guideline for building on what you already have.

Answer 4

You obviously have side effects from some tests that create unintended dependencies. Debug.

Good unit tests are atomic and with zero dependencies to other tests (important). A good practice is that each test creates (removes) everything it is dependent on before running the test. Cleaning up afterwards is also a good practice, it really helps and is recommended but not 100 % necessary.