I'm just beginning to evaluate mocking frameworks for my team and am wondering if anyone has any pointers to reference documentation or experience that you can share regarding the cost of mocking when doing performance tests.
Links? Personal experience? Details appreciated.
IIRC TypeMock uses the Profiler API, which generally does add quite a bit of overhead, but should still be faster than running the application via a profiler.
NCover also uses the Profiler API, and seems to be quite fast.
Aaron Jensen created a test project and did some performance testing. http://codebetter.com/blogs/aaron.jensen/archive/2008/05/08/mock-framework-benchmarks.aspx
I'd rather choose based on API and capabilities though, but performance can be an issue with TDD and running your tests loads of times.
We have been using TypeMock for a couple of years and in my experience there is no significant overhead in performance (I'm sure there is an overhead, it is just not a big issue).
However, due to the nature of how TypeMock works there are several things to consider. As TypeMock basically works by injecting code on the fly the errors can sometimes be very exotic. Reporting errors can thus become a bit of a daunting task. Be prepared to dig into IL.
My experience is that it can be hard to explain the "average developer" how TypeMock works. It quickly becomes complicated and even though their Trace tools make troubleshooting doable it still leaves a bit of a support task.
Also, as TypeMock will let you mock anything (except for mscorlib), you don't really need to add the necessary levels of indirection to your code. This is a feature and TypeMock is not really at fault here. Still, I've seen a lot of developers trying to solve their problems by mocking all over the place instead of decoupling the code. That doesn't improve the overall code quality IMO.
I have been testing mocking frameworks (Moq and TypeMock specifically). TypeMock is much more powerful and flexible but because it plugs into the framework as a profiler it really does have a signficant impact on performance.
My conclusion is that TypeMock is an excellent tool for non-load test scenarios. Moq is less flexible... but much lighter weight and does not have a broad effect on general performance. With Moq you have to setup your applications specifically to be able to mock out the external dependencies (an excercise in good design anyway) but has proven to be a much better fit for my load related scenarios.
Unfortunately I did not record actual numbers in my tests regarding Moq vs TypeMock but the performance benefit of Moq is signficant in my experience.
Most people don't feel the performance pain in their unit tests - or, at least there's a fair chance that someone's tests are slow not because of the mocking framework he is using.
However, Mocking Frameworks Compare project has some performance comparison you can run on your own, look here