MSTest executing all my tests simultaneously breaks tests - what to do

Question

Ok, this is annoying.

MSTest executes all of my tests simultaneously which causes some of them to fail. No this is not because my tests are fragile and susceptible to build order rather it is because this is a demo project in which I use a Db4o object database running from a file.

So I have a couple of DataAccess tests checking that my repositories work correctly and boom, MSTest blows up. Since it tries to run all its tests at the same time it gets an error when a test tries to access the database file while other tests are using it.

Can anyone think of a quick way around this? I don't want to ditch MSTest (ok I do but another story) and I sure as heck don't want to run a full-blown database service so I'll take any way to force MSTest not to run simultaneously or tricks with opening files.

Anyone have any ideas?

Answer 1

You might want to try using a Monitor and entering in TestInitialize and exiting on TestCleanup. If your test classes all depend on the external file, you'll need to use a single lock object for all of them.

public static class LockClass
{
    public static object LockObject = new object();
}

...

[TestInitialize]
public void TestSetup()
{
     Monitor.Enter(LockClass.LockObject);
}

[TestCleanup]
public void TestCleanup()
{
     Monitor.Exit(LockClass.LockObject);
}

This should force all of your tests to run serially and as long as all of your tests pass/fail they should run. If any of them throws an unexpected exception, though, all the rest will hang since the Exit code won't be run for the test that blows up.

Answer 2

I had a try using locks in this manner. What I experienced, however, was that VS2010 does not execute the tests in parallel by default, but executes them sequencially, in a single thread. (parallel execution could be switched on, however. But this would not prevent the problem completely)

What I find very disturbing is, that the sequencial execution will take place in arbitrary order, even across test classes!

So for example an execution order may look like this:

• Class A - TestInitialize: Lock will be established
• Class A - TestMethod1: Will execute, OK
• Class B - TestInitialize: Lock will be established => Thread will be blocked => Complete UnitTests will be blocked! The cause is that there are no other Threads which would go on executing methods of Class A. So the Montor.Exit() will never be reached.

I do not understand why M$ is doing so. Other UnitTest frameworks (e.g. JUnit) execute the test methods class-wise. Otherwise there will be some interleaving of SetUp/TearDown method which would cause the chaos described...

Is there anybody out there knowing how to prevent MSTest jumping between test classes? (Currently I use Resharpers test runner, which behaves as expected, executing all tests methods of one classe before proceeding with the next class)