How do I run a .NET Unit Test repeatedly with a set of parameters cartesianed

Question

I have 1 unit test method that needs several parameters. I would like to run this test once for every possible value of a cartesian with a predefined list for each parameter. I imagine that parameters could be passes in via the test context, but I do not want to connect to an external database.

For example: If I had 2 parameters with the following possible values, the test would execute 6 times (order doesn't matter). (pseudo-code)

p1 = { 1, 5, 10 }
p2 = { "blue", "red" }

test 1: ( 1, "red" )
test 2: ( 5, "red" )
test 3: ( 10, "red" )
test 4: ( 1, "blue" )
test 5: ( 5, "blue" )
test 6: ( 10, "blue" )

Note: I'm using the built-in Visual Studio 2010 unit testing, not NUnit or one of the many other unit test frameworks.

Edit:

I'm storing the possible values as enumerations in the test class, but it is also reasonable to use arrays. My goal is to automate the link between the source enumerations/arrays and the actual test. While my sample only has 2 parameters with 6 permutations, the actual set is much larger. I do not want to skip a scenerio just because I missed something in a manual conversion.

Answer 1

The answer to your question is: You can use "Data-driven tests". You can store your values in comma-seperated-value files, XML files, or SQL database files. I've been doing this with VS2k8, so your process with VS2k10 may be a little bit different.

First, create a file with your test data. If you create a CSV file, it might look like

param1, param2
1, blue
1, red
5, blue
5, red
10, red
10, blue

Go to "Test View", select the test you want to use the data with. In properties, click the ellipsis dots next to "Data Connection String". Specify your file.

Now, in your unit test code, you will specify the data from your file as testContext.DataRow("param1") and testContext.DataRow("param2").

When you run the test, you will get a test result for each data row the test runs with. How convenient!

Update If you want to automatically use Cartesian products of parameters that are kept as enumerations, you might want to use nested for loops like

dim testResults as dictionary(of triplet,string)
for each x as foo in fooCollection
for each y as bar in barCollection
for each z as foobar in foobarCollection Try ' test code
testResults.add(new triplet(x,y,z),"PASS")
catch ex as exception
testResults.add(new triplet(x,y,z),"ERROR: " + ex.toString())
End try

Answer 2

For this type of testing, definitely consider Pex over rolling your own.

However, why not create an outer test which generates all the combinations from your sets at runtime and calls the test method once for each combination?

Answer 3

As @hemp mentioned, you can use an outer test that generates all the permutations of your enums. The class at http://www.codeproject.com/KB/recipes/Combinatorics.aspx looks pretty interesting.

However, is testing all possible combinations just a sanity check? To me it's a code smell, and I would look for edge cases instead of bogging down test run time with running the permutations. Just a thought. :)

Answer 4

@Rising Star's idea of using a data driven was the best solution. I used that in conjunction with a T4 script to generate a CSV containing the cartesianed enums. Here is the script for your reference (uses the T4 Toolbox).

<#@ template debug="false" hostspecific="true" language="C#" #>
<#@ output extension=".csv" #>
<#@ include file="T4Toolbox.tt" #>
<#@ import namespace="EnvDTE" #>
<#@ import namespace="System.Collections.Generic" #>
<#
    List<CodeEnum> enums = GetEnums("TestClass.cs");    
    bool first = true;

    // Header
    foreach(CodeEnum e in enums)
    {
        if(first)
            first = false;
        else
            Write(",");

        Write(e.Name);
    }
    WriteLine("");

    // Data
    WriteData(enums);
#>
<#+
    private void WriteData(List<CodeEnum> enums)
    {
        WriteData(enums, new string[enums.Count], 0);
    }

    private void WriteData(List<CodeEnum> enums, string[] values, int level)
    {   
        foreach (CodeElement element in enums[level].Children)
        {
            values[level] = element.Name;

            if(level + 1 < enums.Count)
            {
                WriteData(enums, values, level + 1);
            }
            else
            {
                WriteLine(string.Join(",", values));
            }
        }
    }

    private List<CodeEnum> GetEnums(string enumFile)
    {
        ProjectItem projectItem = TransformationContext.FindProjectItem(enumFile);
        FileCodeModel codeModel = projectItem.FileCodeModel;
        return FindEnums(codeModel.CodeElements);
    }

    private List<CodeEnum> FindEnums(CodeElements elements)
    {
        List<CodeEnum> enums = new List<CodeEnum>();

        FindEnums(elements, enums);

        return enums.Count == 0
            ? null
            : enums;
    }

    private void FindEnums(CodeElements elements, List<CodeEnum> enums)
    {
        foreach (CodeElement element in elements)
        {
            if (element is CodeEnum)
                enums.Add((CodeEnum)element);

            FindEnums(element.Children, enums);
        }
    }
#>