Multithread a unit test

Question

In order to test concurrency issues, I want to call the same method on two different threads at exactly the same time using a unit test. In fact I probably want to call the same method on several threads at the same time.

I'm using Microsoft's built in unit tester for VS2008.

My thoughts are that I would lock an object and then synchronously setup each of the threads which would immediately wait on the lock and then release the lock which would allow all threads to access the shared resources and test the ability of the code to correctly handle multiple threads.

Has anybody done this before? Am I taking a reasonable approach here?

Answer 1

The problem with this kind of approach is that the results are not reproducible. So, if there was a bug in your code, calling the same test multiple times would result in (for instance) a deadlock in some cases, while in other cases it would not.

Answer 2

I have done this before and even though the results are not always reproducible, they can highlight problems or errors in your code.

I don't use Microsoft's unit testing for it though, I prefer MbUnit which has the ThreadedRepeat attribute which allows you to specify how many threads you want to run. It make the whole multi-threaded test thing much easier, just write a test and add an attribute.

Answer 3

For a unit test I think this is suboptimal as Kibbee indicated it may or may not succed. You'd need to run this hundreds or thousands of times, and even then you may never hit encounter a bug even if a race condition exists.

If your trying to ensure there are no deadlocks you could look at typemock racer: http://www.typemock.com/learn_about_typemock_racer.html I have zero experience with it but it sounds like it can detect a lot of deadlocks.

Answer 4

If you use unit tests to test multi-threaded code, you're probably doing it wrong.

For starters, you don't have any guarantees that both methods will run at a same time; this means that your tests won't guarantee that your code works.

You probably could test if your method works fine in multithreaded environment by inserting sleep commands, but that's too intrusive for unit tests; it's actually a proof of concept and not something I'd leave as unit test after I'm sure that some part of code works as intended.

Multi-threaded tests themselves are fine, to speed up unit testing when unit tests take a lot of time, but unit testing is essentialy for testing small isolated methods.

What you might consider is stress testing, but that's done "from the outside", as opposed to "from the inside" like unit tests. This means that you write a client againt your server and run it thousands of times from multiple machines, testing the whole system instead of the single isolated method. There are two main advantages to stress testing. First, stress testing is used to find exactly the type of defect which happens once in hundred runs in a real-time situation, and second, since they're done "from the outside", they better reproduce the way threads will thread through your application.

Unit tests are in my opionion as bad for testing multi-threaded code as running your code through debugger - while one thread is waiting for you to initiate next step, other thread will time out, and you'll never have the real-life repro.

To summarize, not everything should be unit tested; proof-of-concept tests are nice way of being sure that your thread-safe collection is really thread-safe, and stress testing is great for discovering bugs in multi-threaded code.