How do you add tests for multi threaded support?

Question

I have a Java service which now will execute in a batch mode. Multi threaded support is added to the service so for every batch request a thread pool will be dedicated to execute the batch. The question is how do I test this? I have functional tests that pass under the threaded version of the service but, somehow, I feel there must be an idiom for testing this.

Answer 1

There really isn't a "good" way to do this. The best thing I can suggest would be TestNG, which allows you to annotate your test methods and cause them to be executed in n threads concurrently. For example:

@Test(invocationCount=10, threadPool=10)
public void testSomethingConcurrently() {
    ...
}

My TestNG knowledge is rusty at best, but AFAIK that should invoke the testSomethingConcurrently method 10 times concurrently. This is a nice and declarative way to run multi-threaded tests against your code.

You can of course do something similar in JUnit by spawning threads manually in a loop, but that's insanely ugly and difficult to work with. I had to do this once for a project I was working on; those tests were a nightmare to deal with since their failures weren't always repeatable.

Concurrency testing is difficult and prone to frustration, due to its non-deterministic nature. This is part of why there is such a big push these days to use better concurrency abstractions, ones which are easier to "reason about" and convince one's self of correctness.

Answer 2

I agree with Daniel, concurrency testing is indeed very difficult.

I don't have a solution for concurrency testing, but I will tell you what I do when I want to test code involving multi-threading. Most of my testing is done using JUnit and JMock. Since JMock doesn't work well with multiple threads, I use a extension to JMock that provides synchronous versions of Executor and ScheduledExecutorService. These allow me to test code targeted to be run by multiple threads in a single thread, where I'm also able to control the execution flow. As I said before, this doesn't test concurrency. It only checks the behavior of my code in a single threaded way, but it reduces the amount of errors I get when I switch to the multi thread executors.

Anyway I recommend the use of the new Java concurrency API. It makes things a lot more easy.

Answer 3

Usually the things that bring down multithreaded applications are issues of timing. I suspect that to be able to perform unit testing on the full multithreaded environment would require huge changes to the code base to do that.

What you may well be able to do, though, is to test the implementation of the thread pool in isolation.

By substituting the body of the threads with test code you should be able to construct your pathological conditions of locking and resource usage.

Then, unit test the functional elements in a single threaded environment, where you can ignore timing.

While this isn't perfect it does guarantee repeatability which is of great importance for your unit testing. (as alluded to in Daniel Spiewak's answer)