Maven Spring tests fail when run together, but succeed individually (ehcache closed, IllegalTransactionStateException)

Question

We're using Maven/Surefire and Spring/Hibernate transactional tests for a fairly large web application. There are 138 Test* classes, running a total of 1178 tests.

A straightforward "mvn test" will generate 82 errors, the nature of which tend to imply a corrupt application context:

Many of these:
IllegalTransactionStateException: Pre-bound JDBC Connection found!

A few of these:
NoSuchMethodError: org.hibernate.cache.CacheException.(Ljava/lang/Exception;)V

For every failed test, running the test class individually "mvn test -Dtest=TestFailingClass" succeeds. Indeed, using -Dtest=TestClass1,TestClass2,Etc." with various subsets of all my test classes succeeds or fails in different ways. For instance, running only the failing test classes succeeds with 0 errors.

With no apparent means to control the order of classes tested by Surefire, I have a difficult time determining which of my test classes seem to be leaving the context in a bad state.

What I'm looking for is a strategy to help determine what is happening in some kind of deterministic manner. I can certainly see the order of tests run from the log, but I cannot reproduce that order controllably.

And of course, suggestions for what to do about it ...

Answer 1

With no apparent means to control the order of classes tested by Surefire, I have a difficult time determining which of my test classes seem to be leaving the context in a bad state.

Indeed. And running subsets of the tests will produce different results (from an execution order point of view) making it very hard to debug your problem.

But you could maybe use the patch from SUREFIRE-321 (Run tests in alphabetical order) to get better control (check the comments, one of the poster was facing a very similar problem to yours).

Answer 2

Indeed the problem comes from a corrupt Spring application context. One of the early tests is dirtying the context and causing the following tests to error out.

One difficulty is trying to control the order of the tests while discovering the test causing the trouble.

I was able to accomplish that by using Maven's log to find the order of the test classes run, then excluding tests one at a time from the top. Thirty-four tests in, I found the culprit.

It was a test called TestSpringContexts. Adding @DirtiesContext to these tests solve the problem, but it was also solved by removing calls to context.close() from the tests.

I wrote a blog post about the process here, but that's the gist of the matter: http://mojo.whiteoaks.com/2010/04/27/finding-the-test-that-corrupts-the-suite/

Mojo