Is there any "advanced" testing framework for .NET?

Question

I am looking for a testing framework that allows me to have:

1. Different kinds(categories) of tests. I want to be able to just run all "fast" tests or all "slow" tests. I know MSTest allows you to create lists and run then run them, but I find it annoying and distracting. I'd like to just tag the tests with attributes as I am developing them, and then just have something like a combobox that let's me select which kind of tests I wanna run. I also know that if I have slower tests I can sort them with MSTest so I can see when the faster ones are over. That certainly is only a "hack" and as you have more and more tests, it gets a total mess.
2. Run tests sequentially. When I say sequentially I don't mean that they depend on each other, I just mean that as I have to test the GUI I can't have 2 tests running at the same time.
3. Have the option to have several tests depend on each other. This means that if I have a test A that fails, in some situations, I'd like to not even try to run tests B or C.
4. Have some kind of support in Visual Studio for all of this. I currently am always doing CTRL+B/CTRL+R, A (build / run tests). So having to go and look to another app (even if its launched from VS IDE) each 2 minutes doesn't sound like a good solution.

Is there anything that is able to accomplish this? I find it awkward that with all this trend around testing and tdd, the tools at our disposal are still so primitive and raw.

Answer 1

MbUnit

• Categories: you have the [Category] attribute
• Sequentially: most test runners do that by default.
• Test dependencies: you have the [DependsOn] attribute
• VS support: through Gallio, TestDriven.Net or ReSharper.

Answer 2

VS's unit testing does 1, 2 and 4.

Different kinds(categories) of tests.

You can create multiple test lists by adding multiple "test settings" items, each can run different configurations or a different subset of your tests.

Run tests sequentially.

YOU Can control the level of concurrency used, including setting it to 1. (This does require editing an XML file, but is covered on MSDN).

Have some kind of support in Visual Studio for all of this. I

That's the easy one.

Have the option to have several tests depend on each other.

That's the hard one, as TDD best practice is for tests to be independent tools seem to assume this.

Within one class of tests keep flags and immediately exit tests where conditions have not been met.

Answer 3

You should use the Gallio if you have resharper.
It supports any framework you might choose.
It will help your productivity a lot.

Answer 4

Using MSTest (some options to help if the others don't work out)

You can have TestCategories (using the TestCategory attribute) then use the filters at the top of the Test List Editor to pick out what you want.

You can also create ordered test lists by right clicking the project/folder, click new test at the top and it is one of the options. You then add and order the tests in the batch and it will run them sequentially and optionally fail the rest when one in the sequence fails.

My tests have been running sequentially, so as someone else mentioned you may just need to set the concurrency to 1, but I haven't done it myself.

Answer 5

NUnit

1. Categories - You can mark a test fixture with a [Category("SomeName")] and then use the category names to include/exclude them from a test run
2. Sequential - This is the default for most of the xUnit frameworks. Works out of the box
3. Test-dependencies : Look at this question for more info - http://stackoverflow.com/questions/3396055. Currently not supported in NUnit - but look at the comments to see if any of the workarounds work for you
4. VS Integration - this is a given for MS-Test for obvious reasons. For NUnit, you would need to install some plug-ins e.g. I use Resharper 5.1 (commercial paid)

Answer 6

Using MSTest

For 1: Different kinds(categories) of tests

If you own ReSharper you can group tests based on TestCategory in their test running. (highly recommend ReSharper if you don't have it. You can use it free for 30 days to try it out).

If you don't then use the [Priority(int)] attribute, which you can group by in Visual Studios test runner.

Unfortunetly, using priority, you cannot have multiple tags on a single test.