Tests dependent on commonly used functionality with NUnit

Question

Hi,

I have some initialization code to use my API. The initialization may fail and I´d like to test it in an NUnit test.

After the initialization the API may be used. I´m testing the API too, but all my test methods will use the same, common, initialization code.

What I would ideally like is if this behavior:

1. The Initialization test is run.
2. The other tests are run if [1] succeeded.

In all cases where [1] will fail, so will all other tests. But the valuable information is that [1] fails. That's where I most likely will find the problem. It would be nice if the other tests could be marked with ? or something, indicating that they did not execute as functionality they depend on didn't pass the tests.

I know that tests should not be brittle. But I can't get around the fact that the initialization code is necessary for correct execution of other functionality.

This is a more general problem where some functionality depends on other functionality. Where the "other functionality" is far too commonly used to provide any real value by failing all tests depending on it. It would be better if the "other functionality" would be tested separately.

Answer 1

OK here's how I would go about this...

Put the common initialization into a Setup method since its needed for all tests. If initialization throws an error, you'd see

• all Tests in a suite failing (which I have been trained over time to recognize as a hint that maybe setup / teardown has thrown an exception).
• the stacktrace for the failing tests containing the Setup method.

If this is too implicit for you, you may (although I wouldn't recommend it) add an empty test with a good name to the same suite. If that test shows up as green, you can be sure that Setup / common init code has succeeded.

[Test]
public void VerifySetup() {}

Update: Seem like you have a pretty niche requirement. I don't know of any mechanism in NUnit to specify such conditional execution of tests - e.g. Run Test2 thru 10 only if Test1 passes.

Answer 2

Try this:

1. Define the "Initialization test" as a TestCase

2. Make all your other tests a subclass of this TestCase

3. Create a Suite which runs your tests in some specific order so that your "Initialization test" is first.

Answer 3

I think that this pretty clear cut. The problem is that you're having a hard time separating out your API responsibilities. You have two. Initialization of API and API execution. Writing your tests to have this kind of dependency can kill you.

So I would recommend that the API create an initialization object and then various command objects to execute the API. The command objects will be in some kind of store or you could create on the fly.

The API will use a mocked initialization object and it will use mocked command objects.

The Initialization Object really doesn't have any dependencies except for whatever you need to initialize.

The Command objects will need a mocked initialization object.

[EDIT]

There are two way to get the other tests to fail if the initialization test fails

1. Add a private variable to the test case. private isInitialized = false;. Then all your other tests check the state of this variable if not true then fail.

2. Extend your test case with the API class. Add private function that interrogate state of initialization.

The cleaner of the two is 2. The fastest implementation is 1.

IMHO This can be a code smell when you have to couple your tests in such a manner. If as you said it is an integration tests. Why do you have a separate test for initialization. Integration is more along the lines of run some action against your API. So each integration test MUST initialize the API.

You might want to rethink your cascade failure scenario. It might be too much noise at completion of tests.

[EDIT1a]

The only way I can see to satisfy your requirements is to extend NUnit. Specifically look into Test Case Builders and Test Decorators.

The link is dated March 2008, so hopefully it isn't to out of date.

Answer 4

I've been in contact with the NUnit developers. It's not possible at the moment without writing a pretty complex plugin. The feature will turn up somewhere in the 3.x code base but will not appear in 2.5. I will consider writing it, but not for the time being.