How to write an automated test for thread safety

Question

I have a class which is not thread safe:

class Foo { 
    /* Abstract base class, code which is not thread safe */ 
};

Moreover, if you have foo1 and foo2 objects, you cannot call foo1->someFunc() until foo2->anotherFunc() has returned (this can happen with two threads). This is the situation and it can't be changed (a Foo subclass is actually a wrapper for a python script).

In order to prevent unwanted calls I've created the following -

class FooWrapper {
public:
    FooWrapper(Foo* foo, FooWrappersMutex* mutex);
    /* Wrapped functions from Foo */
};

Internally, FooWrapper wraps calls to the Foo functions with the shared mutex.

I want to test FooWrapper for thread safety. My biggest problem is the fact that threads are managed by the operating system, which means I've got less control on their execution. What I would like to test is the following scenario:

• Thread 1 calls fooWrapper1->someFunc() and blocks while inside the function
• Thread 2 calls fooWrapper2->anotherFunc() and returns immediately (since someFunc() is still executing)
• Thread 1 finishes the execution

What is the simplest to test a scenario like this automatically?

I'm using QT on Win32, although I would prefer a solution which is at least cross-platform as QT is.

Answer 1

So far I've written the following code. Sometimes it works and sometimes the test fails, since the Sleep is not enough for running all threads.

//! Give some time to the other threads
static void YieldThread()
{
#ifdef _WIN32
    Sleep(10);
#endif //_WIN32
}

class FooWithMutex: public Foo {
public:
    QMutex m_mutex;
    virtual void someFunc()
    {
        QMutexLocker(&m_mutex);
    }
    virtual void anotherFunc()
    {
        QMutexLocker(&m_mutex);
    }
};

class ThreadThatCallsFooFunc1: public QThread {
public:
    ThreadThatCallsFooFunc1( FooWrapper& fooWrapper )
        : m_fooWrapper(fooWrapper) {}

    virtual void run()
    {
        m_fooWrapper.someFunc();
    }
private:
    FooWrapper& m_fooWrapper;
};

class ThreadThatCallsFooFunc2: public QThread {
public:
    ThreadThatCallsFooFunc2( FooWrapper& fooWrapper )
        : m_fooWrapper(fooWrapper) {}

    virtual void run()
    {
        m_fooWrapper.anotherFunc();
    }
private:
    FooWrapper& m_fooWrapper;
};

TEST(ScriptPluginWrapperTest, CallsFromMultipleThreads)
{
    // FooWithMutex inherits the abstract Foo and adds
    // mutex lock/unlock on each function.
    FooWithMutex fooWithMutex;

    FooWrapper fooWrapper( &fooWithMutex );
    ThreadThatCallsFooFunc1 thread1(fooWrapper);
    ThreadThatCallsFooFunc2 thread2(fooWrapper);

    fooWithMutex.m_mutex.lock();
    thread1.start(); // Should block

    YieldThread();
    ASSERT_FALSE( thread1.isFinished() );

    thread2.start(); // Should finish immediately
    YieldThread();
    ASSERT_TRUE( thread2.isFinished() );

    fooWithMutex.m_mutex.unlock();

    YieldThread();
    EXPECT_TRUE( thread1.isFinished() );
}

Answer 2

You might want to check out CHESS: A Systematic Testing Tool for Concurrent Software by Microsoft Research. It is a testing framework for multithreaded programs (both .NET and native code).

If I understood that correctly, it replaces the operating system's threading libraries with its own, so that it can control thread switching. Then it analyzes the program to figure out every possible way that the execution streams of the threads can interleave and it re-runs the test suite for every possible interleaving.

Answer 3

Instead of just checking that a particular thread is finished or not, why not create a fake Foo to be invoked by your wrapper in which the functions record the time at which they were actually started/completed. Then your yield thread need only wait long enough to be able to distinguish the difference between the recorded times. In your test you can assert that another_func's start time is after some_func's start time and it's completed time is before some_funcs completed time. Since your fake class is only recording the times, this should be sufficient to guarantee that the wrapper class is working properly.

EDIT: You know, of course, that what your Foo object does could be an anti-pattern, namely Sequential Coupling. Depending what it does, you may be able to handle it by simply having the second method do nothing if the first method has not yet been called. Using the example from the Sequential Coupling link, this would be similar to having the car do nothing when the accelerator pedal is pressed, if the car has not yet been started. If doing nothing is not appropriate, you could either wait and try again later, initiate the "start sequence" in the current thread, or handle it as an error. All of these things could be enforced by your wrapper as well and would probably be easier to test.

You also may need to be careful to make sure that the same method doesn't get invoked twice in sequence if an intervening call to another method is required.

Answer 4

When you start multithreading, your code becomes, by definition, non-deterministic, so testing for thread safety is, in the general case, impossible.

But to your very specific question, if you insert long delays inside Foo to cause each Foo method to take a looonnng time, then you can do what you ask. That is, the probablity of the first thread returning before the second thread enters the call becomes essentially zero.

But what is it that you're really trying to accomplish? What is this test supposed to test? If you trying to validate that the FooWrappersMutex class works correctly, this won't do it.

Answer 5

http://www.intel.com/cd/software/products/asmo-na/eng/286406.htm

Intel Threadchecker.

If I recall correctly the tool checks your code for theoretically possible data races. The point is you don't need to run your code to check whether it's correct or not.