Unit Testing Frameworks - What are the key features

Question

I've read a lot about the available .NET unit testing frameworks. What is not clear to me is what are the key features that differentiate the available frameworks. What are the must-have features that you look for when evaluating unit testing frameworks?

Answer 1

They all do much the same thing, and if they don't, they have extension points which mean you can accomplish the same thing. And in that sense, you can evaluate them based on the same criteria you use when choosing any framework (e.g. popularity (NUnit), MS-Certiffied (MS-Test) etc etc)

However, if you want advice on choosing one, I suggest you read up on the XUnit "why did we build xunit" page which points out certain of the issues surrounding unit testing frameworks.

If you want to see some similarities between all the frameworks, you can see from the table here, they all have a lot of similar features, just different words for things.

Of course choosing a BDD framework is a different kettle of fish.

Answer 2

Here are some things I look for

• Speed. Frameworks (and test runners) are not all created equal. If your unit tests lag, your productive time is wasted.
• Asserts. These need to be plentiful to provide for many scenarios. Do you like exceptions caught using an attribute or an Assert.Throws, for example? Are the asserts capable of doing numeric comparisons with a specified tolerance?
• Miscellanea. Useful things that certain frameworks have such as e.g. row tests or being able to read in test data in XML format.

Answer 3

The thing that I look for is toolset support, with my particular priority being Continuous Integration support. The most important part of unit tests (for me at least, not being a huge fan of TDD) is to plug them into my CI build server.

For this reason I favour nUnit, because there are pre-built build tasks for it in CruiseControl, which is my build server of choice. That is not to say that you can't plug any other testing frameworks into CC, nUnit just happens to be an easy one to use.

As for available features, MSTest has some really nice add-ons when you run it with the fantastically priced Visual Studio Team System with TFS in the back-end.

Having said all of that, there really is not that much difference in the functionalty of the individual testing frameworks (I have experience with xUnit, nUnit and MSTest) - they all allow you to define unit tests, and report on how mnay passed and how many failed. They all have some sort of GUI, and it is possible to integrate them all with build servers.

Answer 4

I agree that they are all quite similar when it comes to assertions and other basic features. Where they begin to differ is in more advanced options, such as scripting frameworks, build tools, automated integration, etc.

If you intend to run a highly automated build process using something like CruiseControl.NET with a lot of custom build options and custom scritpting I would probably use NUnit. There are a lot more hooks for triggering other actions based on the test results.

If you are just getting started with unit testing and only plan on doing basic integration testing I would stick with MStest because of its tight integration with Visual Studio 2008. (NUnit has options for integrating with the IDE but the better tools cost money.)

Here is a little comparison table:

          | NUnit                            | MStest
---------------------------------------------------------------------------------
Asserts   | missing some collection asserts  | missing some numeric/date asserts
---------------------------------------------------------------------------------
Speed     | test run fast, faster setup on   | w/ VS 2008 no setup necessary on
          | servers and the like             | the client, test run fairly fast.
---------------------------------------------------------------------------------
Options   | a lot of 3rd party add-ons and   | some 3rd party tools
          | other tools                      |
---------------------------------------------------------------------------------
CC.NET    | good integration, lots of tools  | newer versions of CC.NET support
          | and options                      | MS test out of the box, not as
          |                                  | many add on tools
---------------------------------------------------------------------------------

Answer 5

One thing that I miss in pretty much every testing framework is uniform assertions, by which I mean that tests should look the same regardless of what I am asserting. Whether I do state-based assertions on a value, behavior-based assertions on a mock or asserting that an exception should be thrown.

I am going to use Ruby as an example, but this applies to any environment. (And, BTW, you are not limited to C# frameworks for testing .NET applications, you can use any .NET language, of which Ruby of course is one, as are Python, Scheme, F#, …)

This is test/unit (a port of an early version of JUnit) with Mocha for mocking.

def test_state
  assert_equal 2, 1+1
end

def test_exception
  assert_raises(NoMethodError) { [].not_a_method }
end

def test_behavior
  o = Object.new
  o.expects(:ping).with(:pong)
  o.ping(:pong)
end

Note how all the assertions look different and are in a different place and the actual test code is in three different place as well.

And this is Expectations, which AFAIK is the only framework that gets it right. (Indeed, it was created specifically for the purpose of uniform assertions.) Unfortunately, it is no longer maintained or supported as the author isn't doing any Ruby work anymore.

expect 2 do
  1+1
end

expect NoMethodError do
  [].not_a_method
end

expect Object.new.to.receive(:ping).with(:pong) do |o|
  o.ping(:pong)
end

Note how the assertions always look the same, and both the assertions and the test code are always in the same spot.

Newer frameworks, such as xUnit.Net and Mockito are definitely getting close, though.

Answer 6

Developer-facing unit tests (note that I am not talking about customer-facing acceptance tests!) are like production code: they should be written in such a way that it is obvious what they are doing. Indeed, this is even more important for tests than it is for production code, because with production code you have the tests to tell you whether the code works or not, with tests, all you have is reading them and seeing whether they make sense or not.

Therefore, developer-facing unit tests should not need comments. And test names are comments, therefore, your framework should not force you to name your tests.

Here is an example (in RSpec, which of course can also be used on .NET, this time):

describe Array do
  it 'should be empty' do
    Array.new.should be_empty
  end
end

This is just stupid. Any developer that needs a test name to understand what this test is doing should seriously rethink his career choice. Also, all the same problems with comments apply: what if somebody changes the test but forgets to change the name? Now the name is not only useless, but also misleading! This is much better:

describe Array do
  it do
    Array.new.should be_empty
  end
end

RSpec has another nifty trick up its sleeve: it will automatically create an instance of the class you are testing for you, so that you don't have to say ClassUnderTest.new.should again and again, you can just say should:

describe Array do
  it { should be_empty }
end

And, BTW, just like a developer can figure out what this is testing, so can RSpec. This is what it spits out in all three cases if you ask it for a customer-readable summary of the testsuite:

Array
 - should be empty