why testing an individual junit test works, while testing them together won't?

Question

The test that fails when tested together with mvn test (or through the ide) is called EmpiricalTest.

If I test the file alone it goes through, but not otherwise. Why could that be?

You can checkout the Maven source code (to test) from here.

This is how I make sure the database is 'blank' before each test:

abstract public class PersistenceTest {

@Before
public void setUp() {
    db.destroy();
    assertIsEmpty(MUser.class);
    assertIsEmpty(Meaning.class);
    assertIsEmpty(Expression.class);
}

private <Entity> void assertIsEmpty(final Class<Entity> entityClass){
    final List<Entity> all = db.getAll(entityClass);
    Assert.assertTrue(all.isEmpty());
}

and the test that fails:

public class EmpiricalTest extends PersistenceTest {

Answer 1

This very much sounds as if there are dependencies between the test. As far as I understand from looking at your test, you're accessing the your data storage in the test. Is there a chance that one of the tests doesn't properly cleanup his traces, therefore causing others to fail??

Testing against a DB is usually not considered a unit test, though it is very useful. Those kind of tests (you may call them integration tests) are however more difficult and time consuming to code because you have to pay a lot of attention that your test leaves the environment in the exact state it found it before.

Answer 2

It got to do with the id automatically assigned. The PU creates a SEQUENCE table, and although I empty the database from my entities, I don't actually drop that table. So when I'm testing EmpiricalTest alone the sequence starts as expected from 1, while when testing together the test is executed later and starts with a higher, unexpected number.

This leads to this question.

Answer 3

Your problem is very common. In ideal TDD world each test should be executed in the perfect isolation from the other test. You violated the isolation and that's the problem.

However there is no simple solution for the test isolation problem. The main reason is that SQL DLL doesn't support database creation/deletion, while automatically droping tables is complicated due to the possible complex foreign keys constrains.

In my experience the best idea is to execute tests within transaction and rollback data on the end of the test (just like Pascal suggested). Spring test module provides great support for that.

In case you cannot execute test within the transaction boundaries (like yours) you must be sure that each of your test doesn't leave anything in the database (including foreignkeys, constrains, sequences, etc.) and also that tests are designed to be independent of each others (for example don't depend on autogenerated id value because sequence generation could be executed in previous tests).

You must debug your Maven test session order to check out what is wrong with the assertion (I guess that you cannot tell that from the Surefire logs). Then fix the tests (both the failing one and the other one which leaves the rubbish in the DB) to be isolated from each others.