What are the real-world pros and cons of each of the major mocking frameworks?

views:

612

answers:

+12 Q:

What are the real-world pros and cons of each of the major mocking frameworks?

see also "What should I consider when choosing a mocking framework for .Net"

I'm trying to decide on a mocking framework to use on a .NET project I've recently embarked on. I'd like to speed my research on the different frameworks. I've recently read this blog post http://codevanced.net/post/Mocking-frameworks-comparison.aspx and wondered if any of the StackOverflow audience has anything to add in the way of real-world advantages and caveats to the frameworks.

Could people could list the pros/cons of the mocking frameworks they either currently use or have investigated for their own use on .NET projects. I think this would be not only a help to me to decide for my current project, but it will help others make more informed decisions when picking the correct framework for their situation. I'm not an expert on any of the frameworks but I would like to get arguments for and against the major frameworks I've come across:

RhinoMocks
Moq
TypeMock Isolator
NMock
Moles

And other usable alternatives that I've missed. I'd also like insights from users that have switched or stopped using products because of issues.

+2 A:

So far I have used RhinoMocks and Moq. Moq is currently my favourite due to its simplicity which is currently all I need. RhinoMocks is pretty powerful but I have never been in the position to fully tap into it.

flq 2009-11-11 22:26:39

I started out with RhinoMocks and quickly moved to Moq once I heard about it as well. Moq is just so gosh darn easy to use.

Chris 2009-11-11 22:29:08

You may want to keep in mind that if you need to support a multi-language environment (e.g. VB) all of the code configurable frameworks (I can speak to Moq and RhinoMocks directly) are going to be painful given the (lack of) anonymous delegate/lambda syntax in VB. This will be more possible in Visual Studio 2010/VB 10 but will still not be comparable to the nice C# lambda syntax.

TypeMock appears to have some support for VB

Ben Von Handorf 2009-11-11 23:05:52

I don't necessarily need cross language compatibility, although that would definitely be a nice bonus feature. That might sway me if two frameworks were otherwise very similar, but it's doubtful that I'd trade that for another useful feature.

BenAlabaster 2009-11-12 03:30:59

+2 A:

We've used Rhino Mocks for more than a year now. PRO:

easy to create the mocks
can mock public and internal methods
can mock interfaces or classes
can create partial mocks (mocking only specific methods from a class)

AGAINST:

methods must be at least internal and virtual (can mess with your architecture)
difficult to use for property-by-property asserts, especially for collections of objects that get created inside the test scope - the constraints syntax gets complicated
you have to be careful when the recording stops and the playback begins
careful about what calls are being mocked (like a property call that you didn't see or a method that wasn't virtual) - the errors you may get are not very helpful

As a general note, we've found that using the mocking frameworks promotes "white box" testing (especially for unit tests). We ended up with tests that validated HOW things were done, not WHAT they were doing. They were useless for refactorings and we had to rewrite most of them.

vladhorby 2009-11-12 00:34:31

Personally I prefer doing state testing with stubs to behavior testing with verifiable mocks. This can help avoid fragile tests that validate the "how".

TrueWill 2009-11-12 02:02:22

Ooh, having to design your code to accommodate the mocking frame work is a definite deal breaker for me. +1 for your insights on that, very valuable.

BenAlabaster 2009-11-12 03:28:09

@Ben - you're going to have that restriction with anything except TypeMock. If you're mocking interfaces there's no problem. With concrete classes, you can only mock out virtual methods.

TrueWill 2009-11-12 04:44:48

@TrueWill - Good design principles are there because they make your code simpler, more extensible, easier to maintain etc, not to cater to your mocking framework. Following good design principles might make unit testing easier, but catering to your mocking framework in your design decisions risks compromising good design principles and tightly coupling your code and mocking framework. Isn't dependency something we've been trying to get away from since the dawn of Dependency Injection, IoC and factory patterns? Why is it suddenly okay to start designing our code around *this* dependency?

BenAlabaster 2009-11-12 15:45:32

@Ben - If your code follows the Dependency Inversion Principle, you will not encounter this restriction. It's only when you have concrete dependencies that you can't easily decouple (read: legacy code) that you need TypeMock. You don't need (or want) to deploy your mocking framework; it's only used by your unit tests. Your production code should not be coupled to the mocking framework.

TrueWill 2009-11-12 18:07:15

@TrueWill - I agree, although coupling of production code to the mocking framework for the purpose of deployment wasn't what I meant. What I meant was that if your code is designed around being able to mock your objects with a particular framework rather than using best design practices from a coding standpoint then there is a possibility that it will become harder to switch out your mocking framework for another one if you want to change your testing platform.

BenAlabaster 2009-11-12 18:17:20

@TrueWill - You are right, the Dependency Inversion is what really works. It allows you to stub the real external dependencies of the system like web services and databases and run tests over whole components or the whole system, not just at class level. We've developed a simple in memory "list-based" database that can replace the real database for the tests via dependency injection without affecting the production code. Unit tests at class / method level get in the way of refactorings that extend over several classes.

vladhorby 2009-11-12 18:51:17

@Ben - absolutely. The choice of a mocking/isolation framework should not affect the design of production code.

TrueWill 2009-11-12 19:08:57

+1 A:

Like Frank and Chris, I tried RhinoMocks and switched to Moq. I haven't been disappointed. See my series of blog posts:

EDIT: Note that I generally do state-based testing with stubs; I seldom do behavior testing with verifiable mocks.

TrueWill 2009-11-12 01:55:11

Can you elaborate on what pushed you to switch? Was it the stubbing issues you ran into with Rhino Mocks that caused you to jump ship or was it something that Moq was just doing better?

BenAlabaster 2009-11-12 03:29:25

@Ben - It was a combination of the stubbing issues and the complexity. With Rhino Mocks I felt like I was fighting against the library (and I was always referencing the documentation); Moq seemed much more natural to me. I'd suggest trying both and sticking with the one that most closely matches your idioms.

TrueWill 2009-11-12 04:41:12

+1 A:

I've not used all those frameworks, but I looked at RhinoMocks and Moq, and went with Moq because it feels more elegant and much simpler. I am using the trunk version which includes a must-have fix for the 4 argument limit imposed on callbacks in the most recent 4.0 beta release.

I especially like the default Moq behavior, which doesn't behave like a strict Mock Object failing tests when unexpected calls are made. You can configure it to do this if you want, but I find that requires me spending way too much time setting up expectations and not enough time testing.

anelson 2009-11-12 02:21:52

+7 A:

I don't know Moles at all, but I'll cover the ones I know a bit about (I really need a table for this, though).

Moq

Pros

Type-safe
Consistent interface
Encourages good design

Cons

Not as full-featured as some of its competitors
- It can't mock delegates
- It can't do ordered expectations
- probably other things I can't think of right now...
Can only mock interfaces and virtual/abstract members

Rhino Mocks

Pros

Type-safe
Full feature set
Encourages good design

Cons

Confusing API. There are too many different ways to do the same things, and if you combine them in the wrong way it just doesn't work.
Can only mock interfaces and virtual/abstract members

TypeMock Isolator

Pros

Type-safe (AFAIR)
Can mock anything

Cons

Very invasive
Potential Vendor Lock-In
Does not encourage good design

NMock

Pros

Encourages good design
Works on any version of .NET (even 1.1)

Cons

Not type-safe
Can only mock interfaces and virtual/abstract members

Please note that particularly the advantages and disadvantages regarding TypeMock are highly controversial. I published my own take on the matter on my blog.

I started out with NMock when that was the only option back in 2003, then migrated to Rhino Mocks because of its type safety, and now use Moq because of the simpler API.

Mark Seemann 2009-11-12 08:05:42

As to the ordered expectations in Moq: Phil Haack shows a nice way here to circumvent that: http://haacked.com/archive/0001/01/01/moq-sequences.aspx

flq 2009-11-12 10:54:15

@Mark: Can you elaborate on your observation that TypeMock is very invasive? In what way?

BenAlabaster 2009-11-12 15:04:16

@Mark: P.S. I read your blog post regarding TypeMock and given the arguments raised by Roy Osherove and your responses [and not knowing anything else about TypeMock], I would have to conclude that your argument seems to be more compelling.

BenAlabaster 2009-11-12 15:24:31

@BenAlabaster: It uses the Profiling API to intercept method calls etc. at run-time. Basically, the code that's running is not the code you (or someone else) wrote.

Mark Seemann 2009-11-12 15:26:33

+1 - great answer and great blog post! I'm not clear on what you mean by Moq can't mock delegates, though. It can mock events, and you can always pass a lambda into any method with a delegate parameter (no need for a framework there).

TrueWill 2009-11-12 18:21:39

@TrueWill: Let's say that you have a class that is injected with a delegate instead of a real interface/base class. With Rhino Mocks, you can actually set up expecectations on that and pass a dynamically generated instance. Last time I checked, that wasn't possible in Moq.

Mark Seemann 2009-11-12 18:54:27

@Mark I can think of a bunch of places I pass delegates... so that might make Moq a tough pill to swallow - at least for this project.

BenAlabaster 2009-11-12 20:58:48

@BenAlabaster: Well, in practice it turns out to be not so much of an issue, since you can always just pass a lambda expression instead.

Mark Seemann 2009-11-13 04:21:16

@Mark - That's the real world info I was after; Can't mock delegates but can be imitated by passing lambda expressions instead. That's really useful.

BenAlabaster 2009-11-13 13:34:47

I have to disagree with the "Encourages good design / Does not encourage good design" evaluations.Besides being higly subjective and controversial, this ignores several obvious real-world situations where the use of limited mocking tools can lead to poor code. For example, a tool that is unable to mock a non-virtual method forces either a separated interface to be created (which most likely will be pointless), or methods not designed for subclassing to be declared as virtual; this is not exactly encouraging good OO design...

Rogerio 2010-01-07 14:10:22

@Rogerio: Well, I disagree, although I know it controversial. It often turns out that the separated interface or virtual method you originally thought only made sense for Testability reasons suddenly become handy in the production code. However, it takes skill to design correctly, and not just hodge-podge to support unit tests. Done correctly, though, Testability is just a special case of the Open/Closed Principle: http://blog.ploeh.dk/2009/06/05/TestabilityIsReallyTheOpenClosedPrinciple.aspx

Mark Seemann 2010-01-07 14:46:28

Actually, you are agreeing with me 8^), since you admit that sometimes (you said "It *often* turns out ...") introducing a separated interface or virtual method may NOT "become handy in the production code". In such cases, therefore, you would want to mock the class without having to do those things. And this is technically possible (easy even) to do both in .NET and in Java.

Rogerio 2010-01-07 20:26:55

+1 A:

Hi there.

I use TypeMock since I'm developing on SharePoint. As TypeMock can mock anything, it's proved a valuable resource when unit testing our SharePoint webparts, event recievers, workflows, etc.

On the downside, TypeMock can be expensive, however there is a version available which is specific for SharePoint and costs less then the full TypeMock package. I highly recommend it.

The one thing I do disagree with is this notion that TypeMock does not make you design your code very well. Often the classes I create, and overall code, are designed well. Just because I use TypeMock doesn't mean I sacrifice the quality of my design - I still practise IoC and SRP. Just because TypeMock can mock anything does't mean I write my code to reflect that ability.

Cheers. Jas.

Jason Evans 2009-11-26 09:55:05

ansaurus

tags:

views:

answers:

What are the real-world pros and cons of each of the major mocking frameworks?

related questions