Junit test that creates other tests

Question

Normally I would have one junit test that shows up in my integration server of choice as one test that passes or fails (in this case I use teamcity). What I need for this specific test is the ability to loop through a directory structure testing that our data files can all be parsed without throwing an exception.

Because we have 30,000+ files that that 1-5 seconds each to parse this test will be run in its own suite. The problem is that I need a way to have one piece of code run as one junit test per file so that if 12 files out of 30,000 files fail I can see which 12 failed not just that one failed, threw a runtimeexception and stopped the test.

I realize that this is not a true "unit" test way of doing things but this simulation is very important to make sure that our content providers are kept in check and do not check in invalid files.

Any suggestions?

Answer 1

I'd write one test that read all the files, either in a loop or some other means, and collected all the failed files in a collection of some kind for reporting.

Maybe a better solution would be a TestNG test with a DataProvider to pass along the list of file paths to read. TestNG will create and run one test for each file path parameter passed in.

Answer 2

I think what you want is parameterized tests. It's available if you're using JUnit4 (or TestNG). Since you mention JUnit, you'll want to look at the @RunWith(Parameterized.class) and @Parameters annotations' documentation.

Answer 3

A Junit3 answer: Create a TestSuite, that creates the instances of the TestCases that you need, with each TestCase initialized according to your dynamic data. The suite will run as a whole within a single JVM instance, but the individual TestCases are independent of each other (setUp, tearDown get called, the error handling is correct, reporting gives what you asked for, etc).

The actual implementation can be a bit clumsy, because TestCase conflates the Name of the test with the METHOD to be run, but that can be worked around.

We normally just combine the suite with the dynamic testcases in the same class, and use the suite() method to get the TestSuite. Ant's JUnit task is smart enough to notice this, for example.

public class DynamicTest extends TestCase {
   String filename ;

   public DynamicTest ( String crntFile ) {
      super("testMethod");
      filename = crntFile ;
   }

   // This is gross, but necessary if you want to be able to
   // distinguish which test failed - otherwise they all share
   // the name DynamicTest.testMethod.
   public String getName() {
      return this.getClass().getName() + " : " + filename ;
   }



   // Here's the actual test
   public void testMethod() {
      File f = new File( filename ) ;
      assertTrue( f.exists() ) ;
   }

   // Here's the magic
   public static TestSuite suite() {

      TestSuite s = new TestSuite() ;

      for ( String crntFile : getListOfFiles() ) {
          s.addTest( new DynamicTest(crntFile ) ) ;
      }

      return s ;
   }
}

You can, of course, separate the TestSuite from the TestCase if you prefer. The TestCase doesn't hold up well stand alone, though, so you'll need to have some care with your naming conventions if your tests are being auto-detected.