How much faster is NUnit compared to MSTest

Question

Roy Osherove, author of The Art Of Unit Testing, has commented on a blog that of the many things NUnit supposedly does better, it being much faster is one of them.

My question is how much faster though, if at all? Are we talking an order of magnitude? 10%? 50%?

I'm asking this because for the moment I can't compare the two. I am trying to setup my test project to be in a dual-mode so that I can switch between them. Unfortunately, I am having a problem with NUnit integrating with the latest version of Microsoft Moles, and also NUnit is conflicting with a third party library (appears log4net related).

So far MSTest seems so much easier to use within Visual Studio 2008. All of the version issues and compatibility problems with NUnit (at least for me) is steering me towards choosing MSTest as the framework for the project (though I may keep the dual mode option). Another plus for MSTest is that I can still use most of NUnit asserts with:

using Assert = NUnit.Framework.Assert;
using Is = NUnit.Framework.Is;

But.... If speed is truly much faster in NUnit, then I'd prefer to use it, despite the pain points.

Lastly, has there been any speed improvement in VS2010 for MSTest?

Answer 1

Hi Matt, I have used both MSTest and NUnit (the first more than the latter), and I can't say I noticed any big differences between the two of them regarding speed (don't get me wrong, the difference might be there, I just haven't noticed it).

The reason I choose MSTest is its integration with Visual Studio, as it makes thigns a lot simpler. Additionally, once I had some issues running a couple of tests because the NUnit tests ran in a different thread appartment than MSTests.

I hope this helps. Damian

Answer 2

Well I took the time to remove Microsoft Moles and the production code that dealt with log4net related issues so that I could actually compare the two. Then I ran tests in both MSTest 2008 and NUnit 2.5.2.

What I found out is that MSTest better reports the duration time of each individual test. If the test is fast enough in NUnit it gets logged as being 0 seconds both in the TestResults.xml file and in the GUI when one clicks on the properties of a test. Nonetheless, I tried to compare the sum of all test speeds and in some cases NUnit is faster and in other cases MSTest is faster. When one is faster than the other it's by about 30%.

Now, where NUnit definitely seems faster is the wait time before the unit tests run. When I attach the GUI (or console) to the VS Debugger, and run the test project, it takes about 3-6 seconds for NUnit to launch and load before the tests can execute. With MSTest it takes between 15-20 seconds. For MSTest it doesn't matter if there is just 1 test or 26, this load time seems to be the same. As regards to how these wait times scale as the test project gets bigger, say in the thousands of unit tests, I can't comment on the difference, though I'd be very interested in knowing.

There also seems to be a little bit of delay in MSTest while the tests are running, perhaps to update the results pane. Thus, I suspect when people say that NUnit is much faster than MSTest, it is because of the loading and updating delays, but the actual test execution time appears to be very similar.