Force JUnit to run one test case at a time

Question

I have a problematic situation with some quite advanced unit tests (using PowerMock for mocking and JUnit 4.5). Without going into too much detail, the first test case of a test class will always succeed, but any following test cases in the same test class fails. However, if I select to only run test case 5 out of 10, for example, it will pass. So all tests pass when being run individually. Is there any way to force JUnit to run one test case at a time? I call JUnit from an ant-script.

I am aware of the problem of dependant test cases, but I can't pinpoint why this is. There are no saved variables across the test cases, so nothing to do at @Before annotation. That's why I'm looking for an emergency solution like forcing JUnit to run tests individually.

Answer 1

It seems that your test cases are dependent, that is: the execution of case-X affects the execution of case-Y. Such a testing system should be avoided (for instance: there's no guarantee on the order at which JUnit will run your cases).

You should refactor your cases to make them independent of each other. Many times the use of @Before and @After methods can help you untangle such dependencies.

Answer 2

Excuse me if I dont answer your question directly, but isn't your problem exactly what TestCase.setUp() and TestCase.tearDown() are supposed to solve? These are methods that the JUnit framework will always call before and after each test case, and are typically used to ensure you begin each test case in the same state.

See also the JavaDoc for TestCase.

Answer 3

Your description shows me, that your unit tests depend each other. That is strongly not recommended in unit tests.

Unit test must be independent and isolated. You have to be able to execute them alone, all of them (in which order, it does not matter).

I know, that does not help you. The problem will be in your @BeforeClass or @Before statements. There will be dependencies. So refactor them and try to isolate the problem.

Probably your mocks are created in your @BeforeClass. Consider to put it into the @Before statement. So there's no instance that last longer than a test case.

Answer 4

I am aware of the problem of dependant test cases, but I can't pinpoint why this is. There are no saved variables across the test cases, so nothing to do at @Before annotation. That's why I'm looking for an emergency solution like forcing JUnit to run tests individually.

The @Before statement is harmless, because it is called for every test case. The @Before**Class** is dangerous, because it has to be static.

Answer 5

Your problem is not that JUnit runs all the tests at once, you problem is that you don't see why a test fails. Solutions:

1. Add more asserts to the tests to make sure that every variable actually contains what you think
2. Download an IDE from the Internet and use the built-in debugger to look at the various variables
3. Dump the state of your objects just before the point where the test fails.
4. Use the "message" part of the asserts to output more information why it fails (see below)
5. Disable all but a handful of tests (in JUnit 3: replace all strings "void test" with "void dtest" in your source; in JUnit 4: Replace "@Test" with "//D@TEST").

Example:

assertEquals(list.toString(), 5, list.size());

Answer 6

Congratulations. You have found a bug. ;-)

If the tests "shouldn't" effect each other, then you may have uncovered a situation where your code can enter a broken state. Try adding asserts and logging to figure out where the code goes wrong. You may even need to run the tests in a debugger and check your code's internal values after the first test.

Answer 7

It sounds to me that perhaps it isn't that you are not setting up or tearing down your tests properly (although additional setup/teardown may be part of the solution), but that perhaps you have shared state in your code that you are not aware of. If an early test is setting a static / singleton / shared variable that you are unaware of, the later tests will fail if they are not expecting this. Even with Mocks this is very possible. You need to find this cause. I agree with the other answers in that your tests have exposed a problem that should not be solved by trying to run the tests differently.

Answer 8

You should check your whole codebase that there are no static variables which refer to mutable state. Ideally the program should have no static mutable state (or at least they should be documented like I did here). Also you should be very careful about cleaning up what you write, if the tests write to the file system or database. Otherwise running the tests may leak some side-effects, which makes it hard to make the tests independent and repeatable.

Maven and Ant contain a "forkmode" parameter for running JUnit tests, which specifies whether each test class gets its own JVM or all tests are run in the same JVM. But they do not have an option for running each test method in its own JVM.