Any suggestions on how to write repeatable unit tests for code that may be susceptible to deadlocks and race conditions?
Right now I'm leaning towards skipping unit tests and focusing on stress tests. The problem with that is you can run a stress test 5 times and see five different results.
EDIT: I know its probably just a dream, but if there were a way to control individual threads and cause them to execute one instruction at a time then I might get somewhere.
Take a look at TypeMock Racer (it's in Beta)
edit: actually Alpha
http://www.typemock.com/Typemock_software_development_tools.html
It is usually possible to force foreseen race-conditions and deadlocks by using things like ManualResetEvent to get each thread into the expected state before releasing it - i.e. get thread A to have the lock and wait for a signal... get thread B to request the lock, etc...
However - you might typically write such a test to investigate a suspected bug, to prove when it is fixed and that it doesn't re-surface. You would generally design around race conditions (but test them as best as is pragmatic).
Can't think of a good automated way, but the closest I've come was to writing a unit test that would 'expose' a deadlock was by using breakpoints in addition to a unit test. I simply added some instructions on where to add the breakpoint. There is some manual effort involved, but with them you can always expose your worse case thread schedule.
Perhaps someone has figured a good way to automate this type of functionality? I could imagine automatically running the debugger, breaking one thread at a specific line, letting another one run until a specific condition, then asserting for the unit test.
I don't think looking for race conditions really falls into the realm of unit testing. More-or-less by definition, the only way to test for race conditions is pseudo-randomly. Unless you're willing to go to the effort of formally proving the correctness of your locking strategy, you'll have to do some stress-testing.
You still need to write unit tests, to verify to correctness of the algorithms, rather than the locking strategy.
When stress-testing multi-threaded code, you'll want to test under conditions where you have one CPU per thread, where you have multiple threads sharing the CPU, and where you have more CPUs than threads (if possible).
You can write a lock class that detects potential deadlocks, by looking at the ordering of the lock operations. We do this by having a thread context that all locks register with when they are acquired (can be made a DEBUG only option).
The idea is to create a graph where the nodes represents locks and a directed edge between A and B means 'lock A was being held when lock B was acquired'. Let your program run using normal loads, then check for cycles in the graph. A cycle means there is potential for deadlock even if your code didn't hit it.
Disclaimer: I work at Typemock.
I've seen the comments regarding Typemock Racer and I want to clarify a few things:
Just a quick comment on (that I apparently require 50 reputation to make)
Racer is not a "stress test tool" it uses static and dynamic analysis algorithms to discover Deadlocks in .NET code. This means that in case no Deadlock was found using Racer no deadlock exist (unless we have a bug :)).
quoting from http://en.wikipedia.org/wiki/Deadlock#Detection
Detecting the possibility of a deadlock before it occurs is much more difficult and is, in fact, generally undecidable, because the halting problem can be rephrased as a deadlock scenario.
So Racer can at best do a heuristic search - unless you solved the halting problem ;)
I have previously used artificial delays in the code that are triggered by some parameters in the request. For example one request tells the server to delay write between two writes and another to do them with no delay in between.
A Mark Bessey writes, this is only useful for creating repro, not for discovering the problem.