views:

107

answers:

3

Is there really any difference between NUnit, MSTest, etc.?

For example, can any of them detect what code changed since the last build and only run the affected unit tests?

Do any of them have tight integration into database setup/rollback for integration-style tests?

Do any of them allow for scripting TCP ports or web services to test the communication layer?

+1  A: 

You can use ant to help run only the tests for the code that last changed, but, that defeats the purpose of unit tests, as you may have broken something that isn't in the file you changed, so all the tests should be ran.

You can design tests to test the database or webservice layers, how much you can do is based on your own ability.

There are problems with the design of unit tests, but that is a problem common to many systems, pattern density.

If you want to do a different style of unit tests you can use the pattern density link above to look at a possible solution, using AOP. You could then write your own layer for using scripts for tests.

MS has a database unit test framework that uses TSQL, but it doesn't automatically do rollbacks at the end of the test, but the tests are scripted out.

James Black
I'm looking for something more intelligent than Ant. I would like something that actually understands that "Class A depends on class B and C." That way if A, B, or C changes, the unit tests for class A are rerun. This is possible given our current technology, we just need someone to put the pieces together.
Jonathan Allen
Tests that actually use web services are a royal pain to do by hand. Something that would allow for recording and playback of the SOAP messages would give us for more meaningful tests the the mocks people currently advocate.
Jonathan Allen
I tend to do my webservice unit tests by just calling them directly and looking at the results, as I can also test the server side by having tests, so I am just testing the communication, and any errors, at that point. The current system is flawed, which is why I suggested the article, as it isn't a perfect solution, but I believe it is a good start on challenging assumptions on unit testing.
James Black
A: 

Detecting what code changed since the last build and only running the affected unit tests is part of what Continuous Integration (CI) servers can do for you. James Black is quite correct that you want to run the relevant suite of tests when you make a change, not just one test or one fixture. However, you may have a shared library project with a suite of tests and a dependent application project with its own suite of tests. Changes to the application code aren't going to affect the library code, so you can set up your CI build to run all the tests if the library code changes but only the application tests if the application code changes. (You may still want to do nightly builds.) This also depends on your revision/source control system; it may not be possible depending on the combination of source control and CI software.

Matthew Scharley is also correct in that dependencies are best handled by isolation/mocking frameworks for the purpose of unit tests. Both NUnit and MSTest (as well as most of the other xUnit frameworks) can also be used for full integration tests (ones that use the actual dependencies, such as calling web services).

For a very biased view on NUnit vs. MSTest, see MSBuild, NAnt, NUnit, MSTest, and frustration. For a strict comparison (somewhat biased in the other direction), see Comparing the MSTest and Nunit Frameworks.

From a personal standpoint (pure opinion), the key features I need have been available in NUnit, so I've seen no reason to evaluate MSTest. Visual Studio Team System has very tight integration with MSTest (I saw a nice demo that showed code coverage visually); if my company were using that (very expensive) version of VS, MSTest might be worth considering. One key question would be if we could get it working with our CI server.

TrueWill
I don't see the point in pushing that responsibility onto the continuous integration server. A CI server should be powerful enough to run all the tests in a reasonable amount of time. What we need is something that can be run on a developer's machine that says "Class A changed. B and C depend on A, so run the tests for A, B, and C."
Jonathan Allen
For web services I want something that will look like a real call from the application's perspective, but have a dummy service on the other side with scripted responses, perferably from previously recorded SOAP packets. This service should be started and stopped by the test framework itself so that it can be run within the context of the tests.
Jonathan Allen
+1  A: 

Detecting what code changed since the last build and running only the affected unit tests isn't really the responsibility of the unit testing framework but rather the responsibility of the build software/scripts.

As far as a unit testing framework goes, they are all fairly comparable to each other, although MSTest is the youngest (I believe). As far as I know, none of them have any implicit (built-in) integration to databases at all, although they all support some notion of setup and teardown for test classes and methods.

Any of them will provide the ability to call web services, etc. but as far as scripting things like TCP ports that is best left to isolation or mocking frameworks, not unit testing frameworks.

Scott Dorman
What's the point of a test framework if all it does is say "not my job" whenever someone wants to do anything non-trival? Any idiot can write a framework that just shows red/green lights, I'm looking for something a bit more useful.
Jonathan Allen
@Grauenwolf: The point of a unit test framework is to perform unit and integration tests, not monitor your code repository for changes. Any of the unit test frameworks are very capable at their job and allow you to write extremely non-trivial unit tests. It sounds like you want an "all-in-one" solution that can automatically do everything...such a system doesn't exist. You need to either use the right tools for the job and not try to force a tool to do something it isn't designed for or write your own homegrown solution.
Scott Dorman