What are the challenges you have faced in writing integration tests?

Question

What are the challenges you have faced in writing integration tests? Does writing unit tests help in writing integration tests?

Answer 1

I deal with distributed software systems where different kinds of server-based applications communicate (usually via messaging). So the perennial problem is keeping provisioned and properly configured deployments of all these complex applications running on their respective servers. Preparing for integration testing has significant QA overhead. VMs can and do help. However, there has been the occasion when clustered software using UDP communications (such as JGroups) has not worked correctly on VMs vs when on physical servers.

The other bane of such testing is an appropriate and consistent database context to test with. Very often it is the case that creating testing setup scripts that can establish a data context in the database is as complicated (and error prone) to devise as the software to be tested itself.

None of the testing methodologies of unit testing, continuous integration builds with built in regression test, etc., have done anything at all to make these testing problems any easier to solve. They're useful and valuable but by no means is there any silver bullet.

Answer 2

UnitTesting works in a much more isolated space than integration testing. So although the tools are much the same, there are some different challenges:

There's the issue of data that changes in back-end systems over time. Sometimes you don't really have any influence over these changes, and they can render detailed integration tests useless. Often this results in making "smoke test" like integration tests that really don't assert too much of the content of the back-end systems. Another approach is to use other services to dynamically look for data that has a specific "shape" that matches your test needs. And none of these techniques really do the same as unit-tests; so it's really a fairly different discipline.

I find it fairly common that (in large projects) there's always a subset of people who specialize in the integration tests, while everyone writes unit tests. Arguably everyone writes integration tests too, but there's usually a bit more to keep tabs on which means that the maintenance over time usually becomes more specialized. Everybody does not need to know the peculiarities of your back-ends in 100% detail.

Answer 3

I've run into the most difficulty testing:

1. Singletons (usually not created by me)
2. Objects that create many of their dependencies, causing it to be difficult to provide "test" versions -- this is one reason why IOC is so nice
3. Heavily message-driven asynchronous objects

I had one co-worker who created everything as a singleton. Thus, objects needing to work together would just say XXX x = XXX.INSTANCE; to get the one singleton. Everything being a singleton makes it near impossible to put any of this into a test framework.

Answer 4

Like unit tests also integration tests need a common understanding and awareness about it.

Every change of requirements may change the behavior of the subject under test. Hence your integration test may break.

Every change of code may change the behavior of the subject under test. Hence your integration test may break.

I think of writing integration tests as like sitting in between the system's stakeholders (the people who write requirements) and the developers. If any side is not aware of the scope of integration tests, you will be the guy who will need sort things out. When something goes wrong typical questions which you have to figure out are:

• Did some requirement changed? (Sure, don’t you check your emails? The protocols have been updated to Spec 1.2.)
• Did the code changed? (Sure, I optimized that database access algorithm by introducing caching, but I was not aware that it would break your integration tests.)

Then you will have to face many scope creepers. Integration tests may become the developer’s quality tests (Yeah, I know my unit tests don’t cover much behavior of that class, but the integration tests will do the rest.) or the stakeholder's acceptance tests (Please add tests how the system will behave if we threw invalid orders at it.) or the admin's stress tests (Hey, your integration tests provide a good way to bombard the server with hundreds of requests per seconds!).

Although this is just one challenge, you will face, it's one of the most important one: Create understanding and awareness for the scope of integration tests.