JUnit 4 test suite problems

Question

Hi, I have a problem with some JUnit 4 tests that I run with a test suite.

If I run the tests individually they work with no problems but when run in a suite most of them, 90% of the test methods, fail with errors. What i noticed is that always the first tests works fine but the rest are failing. Another thing is that a few of the tests the methods are not executed in the right order (the reflection does not work as expected or it does because the retrieval of the methods is not necessarily in the created order). This usually happens if there is more than one test with methods that have the same name. I tried to debug some of the tests and it seems that from a line to the next the value of some attributes becomes null.

Does anyone know what is the problem, or if the behavior is "normal"?

Thanks in advance.

P.S.: OK, the tests do not depend on each other, none of them do and they all have the @BeforeClass, @Before, @After, @AfterClass so between tests everything is cleared up. The tests work with a database but the database is cleared before each test in the @BeforeClass so this should not be the problem.

Simplefied example:

TEST SUITE:

import org.junit.BeforeClass;
import org.junit.runner.RunWith;
import org.junit.runners.Suite;
importy testclasses...;

@RunWith(Suite.class)
@Suite.SuiteClasses({ Test1.class, Test2.class })
public class TestSuiteX {
 @BeforeClass
 public static void setupSuite() { System.out.println("Tests started"); }   
 @AfterClass
 public static void setupSuite() { System.out.println("Tests started"); }   
}

TESTS: The tests are testing the functionalily on a server application running on Glassfish.

Now the tests extend a base class that has the @BeforeClass - method that clears the database and login's and the @AfterClass that only makes a logoff. This is not the source of the problems because the same thing happened before introducing this class.

The class has some public static attributes that are not used in the other tests and implements the 2 controll methods.

The rest of the classes, for this example the two extends the base class and does not owerride the inherited controll methods.

Example of the test classes:

    imports....

    public class Test1 extends AbstractTestClass {  
    protected static Log log = LogFactory.getLog( Test1.class.getName() );

    @Test
    public void test1_A() throws CustomException1, CustomException2 {

        System.out.println("text");

        creates some entities with the server api.
        deletes a couple of entities with the server api.

        //tests if the extities exists in the database
        Assert.assertNull( serverapi.isEntity(..) );

    }

}

and the second :

public class Test1 extends AbstractTestClass {

    protected static Log log = LogFactory.getLog( Test1.class.getName() );

    private static String keyEntity;
    private static EntityDO entity;

    @Test
    public void test1_B() throws CustomException1, CustomException2 {

        System.out.println("text");

        creates some entities with the server api, adds one entities key to the static attribute and one entity DO to the static attribute for the use in the next method.
        deletes a couple of entities with the server api.

        //tests if the extities exists in the database
        Assert.assertNull( serverapi.isEntity(..) );

    }

    @Test
    public void test2_B() throws CustomException1, CustomException2 {

        System.out.println("text");

        deletes the 2 entities, the one retrieved by the key and the one associated with the static DO attribute

        //tests if the deelted entities exists in the database
        Assert.assertNull( serverapi.isEntity(..) );

    }

This is a basic example, the actual tests are more complex but i tried with simplified tests and still it does not work. Thank you.

Answer 1

It seems that you built your test suite on the assumption that the order of executing methods is fixed. This is wrong - JUnit does not guarantee the order of execution of test methods, so you should not count on it.

This is by design - unit tests should be totally independent of each other. To help guaranteeing this, JUnit creates a distinct, new instance of your test class for executing each test method. So whatever attributes you set in one method, will be lost in the next one.

If you have common test setup / teardown code, you should put it into separate methods, annotated with @Before / @After. These are executed before and after each test method.

Update: you wrote

the database is cleared before each test in the @BeforeClass

if this is not a typo, this can be the source of your problems. The DB should be cleared in the @Before method - @BeforeClass is run only once for each class.

Answer 2

The situation you describe sounds like a side-effecting problem. You mention that tests work fine in isolation but are dependent on order of operations: that's usually a critical symptom.

Part of the challenge of setting up a whole suite of test cases is the problem of ensuring that each test starts from a clean state, performs its testing and then cleans up after itself, putting everything back in the clean state.

Keep in mind that there are situations where the standard cleanup routines (e.g., @Before and @After) aren't sufficient. One problem I had some time ago was in a set of databases tests: I was adding records to the database as a part of the test and needed to specifically remove the records that I'd just added.

So, there are times when you need to add specific cleanup code to get back to your original state.

Answer 3

Be careful in how you use @BeforeClass to set up things once and for all, and @Before to set up things before each individual test. And be careful about instance variables.

We may be able to help more specifically, if you can post a simplified example of what is going wrong.