In-container testing vs. mock objects for integration testing

Question

In-container testing is often opposed to testing with mock objects. However, as mock objects simply mimic the behavior of the real objects, isn't the in-container testing the only way to really test the system in its' real environment?

As an partial alternative to in-container testing and mock objects, Spring provides the TestContext framework that initializes Spring nicely without needing to start up the actual application container (web application server, in my case). However, this is somewhat limited approach as it only initializes Spring-specific features while the application server -specific features are not supported. So you cannot test everything. Also, as it is not 100% the same as the default WebApplicationContext that is used in real web execution, isn't this approach a bit hackyish? Is it bad?

For in-container testing, there are at least Cactus (outdated), Jeeunit (a very little project) and JBoss Arquillian (still alpha, but looks promising). I don't see any of these projects too widely used, so is there something bad with in-container testing? The main drawback often mentioned with in-container testing is slow execution speed. However, when run in a continuous integration environment and in a relatively small project, this shouldn't be a problem.

To summarize: should we do either in-container or out-container testing and why? Would you feel bad using mock objects or an alternative initialization mechanism (as in Spring TestContext) for your integration tests?

A subnote: I recently asked about categorization of integration test, which might be relevant.

Answer 1

I'd say we should do both in-container and out-container testing. The main problem I've found with testing in-container is it's much more work to automate everything. You can get some cheaper wins with something like Spring's integration testing support, but we shouldn't fool ourselves that this covers everything that testing in deployment container would.

On the topic of mocks, I find that they can play a part in integration testing with Spring, but I'm more likely to use fake/stub services with canned results for integration and in-container testing (which I'd call system testing).

If you're not sure on what the difference is between mocking and stubbing, check out this Martin Fowler article.

Answer 2

However, as mock objects simply mimic the behavior of the real objects, isn't the in-container testing the only way to really test the system in its' real environment?

I think the short answer is yes, but... I think your "categorization of integration test" question is very relevant. Unit and integration tests are both important although the serve different functions.

Unit tests are closely associated with the code, should startup and run very fast, should be run often by developers iterating on the code, and usually use mocks to a high degree. The idea is to test the code in question, not its dependencies or integration points. The problem with making the unit tests all be in-container is that they will be run less often or they will waste too much developer time.

We have isolated our in-container/integration tests elsewhere in another project with dependencies on the code projects. They are designed to closely mimic production configurations -- as much as possible. These tests take longer to setup, run longer, and are more useful to run by something like cruisecontrol. They are run by hand especially when we are closing in on a release or after development has stabilized. Often I fire up the integration tests when I'm going to lunch or a meeting. When an integration test discovers a bug, we try to write a unit test that also demonstrates the bug with the mocks -- this is often hard and can be impossible.

We usually have a couple small in-container tests with the unit tests just to make sure that the spring wiring works or to test some basic functionality but the rest of the integration testing is done in another project.

All this said, there is not a firm difference between the two. Sometimes we move unit tests that just deal with a lot of data and take a long time out to integration tests and sometimes integration tests run fast enough and are valuable enough to be included along with the code.