Unit testing a multithreaded application?

Question

Does anyone have any advice for a consistent way to unit test a multithreaded application? I have done one application where our mock "worker threads" had a thread.sleep with a time that was specified by a public member variable. We would use this so we could set how long a particular thread would take to complete its work, then we could do our assertions. Any ideas of a better way to do this? Any good mock frameworks for .Net that can handle this?

Answer 1

TypeMock (commercial) has a unit testing framework that automatically tries to find deadlocks in multithreaded applications and I think can be set up to run threads with predictable context switching.

I saw a demo this week at a show -- apparently it's in Alpha (called Racer)

http://www.typemock.com/Typemock_software_development_tools.html

Answer 2

Not quite a unit test, but you could write some test code which repeatedly calls the code that will execute on different threads. Trying to create maximal interleaving between threads with a periodic or final consistency check. Of course this approach has the downside of not being reproducable, so you would need to use extensive logging to figure out what went wrong. This approach would best be coupled with unit tests for each threads individual tasks.

Answer 3

Like GUI Unit-testing, this is a bit of a Waterloo for Automated tests. True threads are by definition.. unpredictable.. they'll interfere in ways that can't be pre-determined. Hence writing true tests is difficult if not impossible.

However you have some company with this... I suggest searching through the archives of the testdrivendevelopment yahoo group. I remember some posts in the vicinity.. Here's one of the newer ones. (If someone would be kind enough to strafe and paraphrase.. that would be great. I'm too sleepy.. Need to LO SO)

Answer 4

If you have to test that a background thread does something, a simple technique I find handy is to to have a WaitUntilTrue method, which looks something like this:

bool WaitUntilTrue(Func<bool> func,
              int timeoutInMillis,
              int timeBetweenChecksMillis)
{
    Stopwatch stopwatch = Stopwatch.StartNew();

    while(stopwatch.ElapsedMillis < timeoutInMillis)
    {
        if (func())
            return true;
        Thread.Sleep(timeBetweenChecksMillis);
    }   
    return false;
}

Used like this:

volatile bool backgroundThreadHasFinished = false;
//run your multithreaded test and make sure the thread sets the above variable.

Assert.IsTrue(WaitUntilTrue(x => backgroundThreadHasFinished, 1000, 10));

This way you don't have to sleep your main testing thread for a long time to give the background thread time to finish. If the background doesn't finish in a reasonable amount of time, the test fails.

Answer 5

My advice would be not to rely on unit tests to detect concurrency issues for several reasons:

• Lack of reproducibility: the tests will fail only once in a while, and won't be really helpful to pinpoint the problems.
• Erratic failing build will annoy everybody in the team - because the last commit will always be wrongly suspected for being the cause of the failing build.
• Deadlocks when encountered are likely to freeze the build until the execution timeout is encountered which can significantly slow down the build.
• The build environment is likely to be a single CPU environment (think build being run in a VM) where concurrency issues may never happen - no matter how much sleeping time is set.
• It defeats somehow the idea of having simple, isolated units of validating code.

Answer 6

It's important to test multi-threaded code on a multi-processor machine. A dual-core machine may not be sufficient. I've seen deadlocks occur on a 4 processor machine that did not occur on a dual-core single processor. Then you need to create a stress test based on a client program that spawns many threads and makes multiple requests against the target application. It helps if the client machine is multi-processor as well so there is more load on the target application.

Answer 7

Step one is to realise that often much of the code you need to unit test is orthogonal to the threading. This means that you should try and break up the code that does the work from the code that does the threading. Once that's done you can easily test the code that does the work using normal unit testing practices. But of course, you knew that.

The problem is then one of testing the threading side of the problem but at least now you have a point where this threading interfaces with the code that does the work and hopefully you have an interface there that you can mock. Now that you have a mock for the code that the threading code calls into I find the best thing to do is add some events to the mock (this may mean you need to hand roll your mock). The events will then be used to allow the test to synchronise with and block the threading code under test.

So, for example, lets say we have something really simple, a multi-threaded queue that processes work items. You'd mock the work item. The interface might include a 'Process()' method that the thread calls to do the work. You'd put two events in there. One that the mock sets when Process() is called and one that the mock waits on after it has set the first event. Now, in your test you can start up your queue, post a mock work item and then wait on the work item's "I'm being processed" event. If all you're testing is that process gets called then you can set the other event and let the thread continue. If you're testing something more complex, like how the queue handles multiple dispatch or something then you might do other things (like post and wait for other work items) before releasing the thread. Since you can wait with a timeout in the test you can make sure that (say) only two work items get processed in parallel, etc, etc. The key thing is that you make the tests deterministic using events that the threaded code blocks on so that the test can control when they run.

I'm sure your situation is more complex but this is the basic method that I use to test threaded code and it works pretty well. You can take a surprising amount of control over multi-threaded code if you mock out the right bits and put synchronisation points in.

Here is some more info on this kind of thing, though it's talking about a C++ codebase: http://www.lenholgate.com/archives/000306.html

Answer 8

I don't think that unit tests are an effective way to find threading bugs, but they can be a good way to demonstrate a known threading bug, isolate it, and test your fix for it. I've also used them to test the basic features of some coordinating class in my application, like a blocking queue for example.

I ported the multithreadedTC library from Java to .NET and called it TickingTest. It lets you start up several threads from a unit test method and coordinate them. It doesn't have all the features of the original, but I've found it useful. The biggest thing it's missing is the ability to monitor threads that are started during the test.

Answer 9

I've came across a research product, called Microsoft Chess. Its specifically designed for non deterministic testing of multithreaded applications. Downside so far, is that it is integrated into VS.