Find JUnit 3 tests that are not included in any suite?

Question

If someone writes a new unit test but they forget to add it to a suite, the continuous build will continue to pass happily even if someone later breaks the test. Is there a way to discover JUnit tests that aren't included in a suite?

EDIT
I know I could move to JUnit 4 but then the question becomes, how do I find tests that are missing the @Test annotation?

Answer 1

Move to junit4?

Answer 2

I'd do the following:

1. Use some combination of find and grep to extract a list of supposed test class names; e.g. grep for files that contain extend TestCase.

2. Use some combination of find and grep on the test reports to extract a list of the test classes that were actually run.

3. Sort the two lists and compare them using diff.

There is a risk of false positives and false negatives; e.g. test classes that don't directly extend TestCase might slip through the net. But this approach will give you a quick "first cut" answer.

Answer 3

Using TestSuite.tests() you can produce list of tests (recursively) included in test suites (list1) and using Java reflection you can produce list of all tests (from classes that extend TestCase) (list2).

Then subtract list1 from list2 to produce list of tests not included in any suite.

The classpath should contain all test classes when running this Java program.

Answer 4

I suggest not manually creating test suites. Instead, make sure your build automatically runs all tests that are under your "tests" directory. Ant supports a way to do this with the <junit> element.

You could also create a suite that finds the tests to run by looking for all non-abstract classes that extend TestCase. You could roll your own reflective suite with Reflections or search around for free implementations (see http://stackoverflow.com/questions/2411938/how-do-i-programmatically-run-all-the-junit-tests-in-my-java-application)