JUnit test with dynamic number of tests

Question

In our project I have several JUnit tests that e.g. take every file from a directory and run a test on it. If I implement a testEveryFileInDirectory method in the TestCase this shows up as only one test that may fail or succeed. But I am interested in the results on each individual file. How can I write a TestCase / TestSuite such that each file shows up as a separate test e.g. in the graphical testrunner of eclipse? (Coding an explicit test method for each file is not an option.)

Compare also the question ParameterizedTest with a name in Eclipse Testrunner.

Answer 1

Should be possible in JUnit 3 by inheriting from TestSuite and overriding the tests() method to list the files and for each return an instance of a subclass of TestCase that takes the filename as constructor parameter and has a test method that tests the file given in the constructor.

In JUnit 4 it might be even easier.

Answer 2

What you seem to have is data-driven tests.. NUnit has this in the form of RowTest extensions (originally from MbUnit).

   [RowTest]
   [Row( 1000, 7, 142.85715)]
   [Row( 1000, 0.00001, 100000000)]
   [Row(4195835, 3145729, 1.3338196)]
   public void DivisionTest(double numerator, double denominator, double result)
   {
     Assert.AreEqual(result, numerator / denominator, 0.00001);
   }

Don't know if this was implemented for JUnit in some other name. If you don't find anything, use a collecting parameter as shown below. (Won't still show up in the GUI as different tests though.)

public void testAllFiles()
{
    string sCollectingParam = "";
    // loop for each file  
      // if test condition fails (can't use junit assert as that would bail out on first failure)
          // add to sCollectingParam with a descriptive log with filename

    assertEquals("Some files failed the test", "", sCollectingParam);
}

Usually tests should not take in any parameters. But every rule has exceptions... I'd go for this in case of some sanity/regression testing. Otherwise a distinct test case for each significantly different dataset is what I'd recommend.

Answer 3

Take a look at Parameterized Tests in JUnit 4.

Actually I did this a few days ago. I'll try to explain ...

First build your test class normally, as you where just testing with one input file. Decorate your class with:

@RunWith(Parameterized.class)

Build one constructor that takes the input that will change in every test call (in this case it may be the file itself)

Then, build a static method that will return a Collection of arrays. Each array in the collection will contain the input arguments for your class constructor e.g. the file. Decorate this method with:

@Parameters

Here's a sample class.

@RunWith(Parameterized.class)
public class ParameterizedTest {

    private File file;

    public ParameterizedTest(File file) {
     this.file = file;
    }

    @Test
    public void test1() throws Exception { }

    @Test
    public void test2() throws Exception { }

    @Parameters
    public static Collection<Object[]> data() {
     // load the files as you want
     Object[] fileArg1 = new Object[] { new File("path1") };
     Object[] fileArg2 = new Object[] { new File("path2") };

     Collection<Object[]> data = new ArrayList<Object[]>();
     data.add(fileArg1);
     data.add(fileArg2);
     return data;
    }
}

Also check this example

Answer 4

JUnit 3

public class XTest extends TestCase {

    public File file;

    public XTest(File file) {
     super(file.toString());
     this.file = file;
    }

    public void testX() {
     fail("Failed: " + file);
    }

}

public class XTestSuite extends TestSuite {

    public static Test suite() {
     TestSuite suite = new TestSuite("XTestSuite");
     File[] files = new File(".").listFiles();
     for (File file : files) {
      suite.addTest(new XTest(file));
     }
     return suite;
    }

}

JUnit 4

import org.junit.Test;
import org.junit.runner.RunWith;
import org.junit.runners.Parameterized;
import org.junit.runners.Parameterized.Parameters;

@RunWith(Parameterized.class)
public class TestY {

    @Parameters
    public static Collection<Object[]> getFiles() {
     Collection<Object[]> params = new ArrayList<Object[]>();
     for (File f : new File(".").listFiles()) {
      Object[] arr = new Object[] { f };
      params.add(arr);
     }
     return params;
    }

    private File file;

    public TestY(File file) {
     this.file = file;
    }

    @Test
    public void testY() {
     fail(file.toString());
    }

}