Is there a way to make Eclipse run a JUnit test mulitple times until failure?

Question

We occasionally have bugs that appear once in every X runs. Before people check in stuff (where it is automatically JUnit'd), our devs need to pass JUnit locally via Eclipse.

Is there some convenient way (built in or high-quality Plugin) to make Eclipse run the same test X times and stop if there's a failure? An alternative to just clicking Run X times?

Note that I'm looking for something in the UI (e.g., right click and say "Run X times" instead of just "Run").

Answer 1

I don't believe there's a built in way for junit to do exactly what you're asking for.

If multiple runs produce different result, you should have a unit test testing that case. Wich might be as simple as running a for loop in the relevant test cases.

Answer 2

If the for loop works, then I agree with nos.

If you need to repeat the entire setup-test-teardown, then you can use a TestSuite:

1. Right-click on the package containing the test to repeat
2. Go to New and choose to create a JUnit test SUITE
3. Make sure that only the test you want to repeat is selected and click through to finish.
4. Edit the file to run it multiple times.

In the file you just find the

addTestSuite(YourTestClass.class)

line, and wrap that in a for loop.

I'm pretty sure that you can use addTest instead of addTestSuite to get it to only run one test from that class if you just want to repeat a single test method.

Answer 3

I know it doesn't answer the question directly but if a test isn't passing every time it is run it is a test smell known as Erratic Test. There are several possible causes for this (from xUnit Test Patterns):

• Interacting Tests
• Interacting Test Suites
• Lonely Test
• Resource Leakage
• Resource Optimism
• Unrepeatable Test
• Test Run War
• Nondeterministic Test

The details of each of these is documented in Chapter 16 of xUnit Test Patterns.

Answer 4

If you really want to run a test class until failure, you need your own runner.

@RunWith(RunUntilFailure.class)
public class YourClass {

    // ....

}

which could be implemented as follows...

package com.example;

import org.junit.internal.runners.*;
import org.junit.runner.notification.*;
import org.junit.runner.*;

public class RunUntilFailure extends Runner {

    private TestClassRunner runner;

    public RunUntilFailure(Class<?> klass) throws InitializationError {
        this.runner = new TestClassRunner(klass);
    }

    @Override
    public Description getDescription() {
        Description description = Description.createSuiteDescription("Run until failure");
        description.addChild(runner.getDescription());
        return description;
    }

    @Override
    public void run(RunNotifier notifier) {
        class L extends RunListener {
            boolean fail = false;
            public void testFailure(Failure failure) throws Exception { fail = true; }
        }
        L listener = new L();
        notifier.addListener(listener);
        while (!listener.fail) runner.run(notifier);
    }

}

...releasing untested code, feeling TDD guilt :)