Easy way of running the same junit test over and over?

Question

Like the title says, I'm looking for some simple way to run JUnit 4.x tests several times in a row automatically using Eclipse.

An example would be running the same test 10 times in a row and reporting back the result.

We already have a complex way of doing this but I'm looking for a simple way of doing it so that I can be sorta sure that the flaky test I've been trying to fix stays fixed.

An ideal solution would be an Eclipse plugin/setting/feature that I am unaware of.

Answer 1

I've found that Spring's repeat annotation is useful for that kind of thing:

@Repeat(value = 10)

http://static.springsource.org/spring/docs/2.5.6/reference/testing.html http://static.springsource.org/spring/docs/2.5.x/api/org/springframework/test/annotation/Repeat.html

Answer 2

Does running a test 'n' times really indicate that you've fixed this problem ?

So something in your code is non-deterministic. Can you mock that out ? Can you extract the functionality unrelated to the non-deterministic code and unit test that ? Can you remove the uncertainty altogether (e.g. if it's threading related, can you make use of queues etc.)

Otherwise you can run your test 10 times, and still have no confidence in it.

Answer 3

The easiest (as in least amount of new code required) way to do this is to run the test as a parametrized test (annotate with an @RunWith(Parameterized.class) and add a method to provide 10 empty parameters). That way the framework will run the test 10 times.

This test would need to be the only test in the class, or better put all test methods should need to be run 10 times in the class.

Here is an example:

@RunWith(Parameterized.class)
public class RunTenTimes {
    @Parameterized.Parameters
    public static List<Object[]> data() {
        return Arrays.asList(new Object[10][0]);
    }

    public RunTenTimes() {
    }

    @Test
    public void runsTenTimes() {
        System.out.println("run");
    }
}

EDIT: With the above, it is possible to even do it with a parameterless constructor, but I'm not sure if the framework authors intended that, or if that will break in the future.

EDIT (in response to comment): If you are implementing your own runner, then you could have the runner run the test 10 times. If you are using a third party runner, then with 4.7, you can use the new @Rule annotation and implement the MethodRule interface so that it takes the statement and executes it 10 times in a for loop. The current disadvantage of this approach is that @Before and @After get run only once. This will likely change in the next version of JUnit (the @Before will run after the @Rule), but regardless you will be acting on the same instance of the object (something that isn't true of the Parameterized runner). This assumes that whatever runner you are running the class with correctly recognizes the @Rule annotations. That is only the case if it is delegating to the JUnit runners.

If you are running with a custom runner that does not recognize the @Rule annotation, then you are really stuck with having to write your own runner that delegates appropriately to that Runner and runs it 10 times.

Note that there are other ways to potentially solve this (such as the Theories runner) but they all require a runner. Unfortunately JUnit does not currently support layers of runners. That is a runner that chains other runners.

Answer 4

Anything wrong with:

@Test
void itWorks()
{
    // stuff
}

@Test
void itWorksRepeatably()
{
    for (int i = 0; i < 10; i++)
    {
        itWorks();
    }
}

Unlike the case where you are testing each of an array of values, you don't particularly care which run failed.

No need to do in configuration or annotation what you can do in code.

Answer 5

There's an Intermittent annotation in the tempus-fugit library which works with JUnit 4.7's @Rule to repeat a test several times or with @RunWith.

For example,

@RunWith(IntermittentTestRunner.class)
public class IntermittentTestRunnerTest {

   private static int testCounter = 0;

   @Test
   @Intermittent(repition = 99)
   public void annotatedTest() {
      testCounter++;
   }
}

After the test is run (with the IntermittentTestRunner in the @RunWith), testCounter would be equal to 99.