How can I pay only once to provide the same expensive data source to multiple unit tests?

Question

G'day!

I have 4.6GB of test data in 220,000 files. A bulk test requires finding, matching, and unzipping pairs of the files. I want to run multiple unit tests across each pair. It'll take a dreadfully long time if I can't persuade the test framework to find, match, and unzip the test data only once regardless of the number of unit tests needing that data.

NUnit's TestDataSource seemed appropriate for just one unit test, but seems to run depth-first with multiple unit tests: each test runs through the entire data set, one test after the other.

Is there a test framework that can run breadth-first, loading each datum once and providing it to multiple unit tests?

What other approach could I try that will give me all of the test results for each datum, not just a pass or the first failing result?

Answer 1

You could put all your tests into one class, and use the test fixture setup (http://www.nunit.org/index.php?p=fixtureSetup&r=2.2.10) to do the long setup, and then all your tests are using the same data. The only problem is if any of your tests change the data then it can mess up the next test, as you shouldn't design for your tests to run in a certain order.

Answer 2

I doubt that there is a simple out of the box solution. The normal pattern is just to run one unit test after the other, maybe feeding multiple inputs into each test.

In your case I would try something like the following.

[TestMethod]
public void MyTest()
{
    Action<Data>[] subtests = GenerateArrayOfDelegatesForAllSubtests();

    while (this.TestDataAvailiable())
    {
        Data data = this.GetNextTestDataSet();

        foreach (Action<Data> subtest in subtests)
        {
            try
            {
                subtest(data);
            }
            catch (Exception exception)
            {
                this.RecordFailedSubtestResult(subtest, data, exception);
            }
        }
    }

    this.UseRecordedSubtestResultsToFailTestWithMeaningfulMessage();
}

This is a lot of custom work, but I am not aware ant cannot think of a better solution at the moment. Hope this helps.