NUnit vs. MbUnit vs. MSTest vs. xUnit.net

Question

There are quite a lot of unittesting frameworks out there for .NET. I found this little feature comparison: http://www.codeplex.com/xunit/Wiki/View.aspx?title=Comparisons

Now I am to choose the best one for us. But how? Does it matter? Which one is most future proof and has a decent momentum behind it? Should I care about the features? While xUnit seems to be most modern and specifically designed for .NET, nUnit again seems to be the one that is widely accepted. MSTest again is already integrated into Visual Studio ...

Answer 1

NUnit is probably the most supported by the 3rd party tools. It's also been around longer than the other three.

I personally don't care much about unit test frameworks, mocking libraries are IMHO much more important (and lock you in much more). Just pick one and stick with it.

Answer 2

It's not a big deal, it's pretty easy to switch between them. MSTest being integrated isn't a big deal either, just grab testdriven.net.

Like the previous person said pick a mocking framework, my favourite at the moment is Moq.

Answer 3

I wouldn't go with MSTest. Although it's probably the most future proof of the frameworks with Microsoft behind it's not the most flexible solution. It won't run stand alone without some hacks. So running it on a build server other than TFS without installing Visual Studio is hard. The visual studio test-runner is actually slower than Testdriven.Net + any of the other frameworks. And because the releases of this framework are tied to releases of Visual Studio there are less updates and if you have to work with an older VS you're tied to an older MSTest.

I don't think it matters a lot which of the other frameworks you use. It's really easy to switch from one to another.

I personally use XUnit.Net or NUnit depending on the preference of my coworkers. NUnit is the most standard. XUnit.Net is the leanest framework.

Answer 4

It's not a big deal on a small/personal scale, but it can become a bigger deal quickly on a larger scale. My employer is a large Microsoft shop, but won't/can't buy into Team System/TFS for a number of reasons. We currently use Subversion + Orcas + MBUnit + TestDriven.NET and it works well, but getting TD.NET was a huge hassle. The version sensitivity of MBUnit + TestDriven.NET is also a big hassle, and having one additional commercial thing (TD.NET) for legal to review and procurement to handle and manage, isn't trivial. My company, like a lot of companies, are fat and happy with a MSDN Subscription model, and it's just not used to handling one off procurements for hundreds of developers. In other words, the fully integrated MS offer, while definitely not always best-of-bread, is a significant value-add in my opinion.

I think we'll stay with our current step because it works and we've already gotten over the hump organizationally, but I sure do wish MS had a compelling offering in this space so we could consolidate and simplify our dev stack a bit.

Answer 5

I know this is an old thread, but I thought I'd post a vote for xUnit.NET. While most of the other testing frameworks mentioned are all pretty much the same, xUnit.NET has taken a pretty unique, modern, and flexible approach to unit testing. It changes termonilogy, so you no longer do you define TestFixtures and Tests...you specify Facts and Theories about your code, which integrates better with the concept of what a test is from a TDD/BDD perspective.

xUnit.NET is also EXTREMELY extensible. Its FactAttribute and TraitAttribute attribute classes are not sealed, and provide overridable base methods that give you a lot of control over how the methods those attributes decorate should be executed. While xUnit.NET in its default form allows you to write test classes that are similar to NUnit test fixtures with their test methods, you are not confined to this form of unit testing at all. You are free to extend the framework to support BDD-style Concern/Context/Observation specifications, as depicted here.

xUnit.NET also supports fit-style testing directly out of the box with its Theory attribute and corresponding data attributes. Fit input data may be loaded from excel, database, or even a custom data source such as a Word document (by extending the base data attribute.) This allows you to capitalize on a single testing platform for both unit tests and integration tests, which can be huge in reducing product dependencies and required training.

Other approaches to testing may also be implemented with xUnit.NET...the possabilities are pretty limitless. Combined with another very forward looking mocking framework, Moq, the two create a very flexible, extensible, and powerful platform for implementing automated testing.

Answer 6

Nunit doesnt work well with mixed-mode projects in C++ so I had to drop it

Answer 7

I personally don't like xUnit.Net because its adoption of a confusing naming scheme such as "Fact", "Theory" etc, instead of the common jargon. There are numerous examples of such (the chart shows them).

Besides the confusion, it makes migration from one framework to another a Replace All hassle. I really wonder which fundamental problem they have addressed with those changes.

Answer 8

Eric, what you used instead of nUnit. I have a mixed-mode project too

Answer 9

Since we have some experience with both MBUnit v3 and NUnit 2.5.x now, I can compare both:

• We found MBUnit less stable than NUnit. Sometimes MBUnit processes don't shut down and prevent CC.Net from cleaning up test folders, causing faux failures.
• Both are equally easy to use for writing unit tests. Semantics are almost intact.
• MBUnit client Gallio Icarus is too slow and sluggish. Example: Its startup time is 30 seconds on my quad core, RAID, 4gb machine, before loading tests. (EDIT: This turned out to be a bug with Gallio which can be worked around by removing all permissions from Gallio.Utility.exe)
• MBUnit test run times on GUI client are much longer than NUnit GUI client. In overall, the GUI client looks bloated.
• Console clients have similar execution times.
• MBUnit supports parallelization of tests though, which can compensate for slow GUI-client execution but that requires manual tuning.
• MBUnit does not support NUnit's Assert.That(output, Is.Condition(expectedValue)) syntax, which makes migration from NUnuit harder if you're used to it (because expectedValue and output needs to be swapped).
• Debugging MBUnit tests is easier because of client's VS integration support.