If a project has 100% unit test coverage, are integration tests still needed?

Question

If a project has 100% unit test coverage, are integration tests still needed?

I have never worked on a project with 100% unit test coverage, but I'm wondering if your project obtains this (or in the 90%), was your experience that you still needed integration tests? (did you need less?)

I ask because integration tests seem to suck. They are often slow, fragile (break easily), opaque (when broken someone has to dive through all the layers to find out what is wrong) and are causing our project to slow way down... I'm beginning to think that having only unit tests (and perhaps a small handful of smoke tests) is the way to go.

In the long run, it seems like integration tests (in my experience) cost more than they save.

Thanks for your consideration.

Answer 1

Yes, besides there are a few different types of code coverage

from wiki:

• Function coverage - Has each function in the program been executed?
• Statement coverage - Has each line of the source code been executed?
• Decision coverage (also known as Branch coverage) - Has each control structure (such as an if statement) evaluated both to true and false?
• Condition coverage - Has each boolean sub-expression evaluated both to true and false (this does not necessarily imply decision coverage)?
• Modified Condition/Decision Coverage (MC/DC) - Has every condition in a decision taken on all possible outcomes at least once? Has each condition been shown to affect that decision outcome independently?
• Path coverage - Has every possible route through a given part of the code been executed?
• Entry/exit coverage - Has every possible call and return of the function been executed?

Path coverage for example, just because every method has been called, doesn't mean that errors wont occur if you call various methods in a given order.

Answer 2

Yes.

Even if all "units" do what they are supposed to do, it is no guarantee that the complete system works as designed.

Answer 3

Unit tests are different from integration tests.

Just to make a point: if I have to choose, I would dump unit tests and go with integration tests. Experience tells that unit tests help to ensure functionality and also find bugs early in the development cycle.

Integration testing is done with product looking close to what it would look to end users. That is important too.

Answer 4

Unit tests are generally all about testing your class in isolation. They should be designed to ensure that given specific inputs your class exhibits predictable and expected behaviors.

Integration tests are generally all about testing your classes in combinations with each other and with "outside" programs using those classes. They should focus on ensuring that when the overall product uses your classes it is doing so in the correct manner.

Answer 5

Yes because the functionality of your software depends on how it's different piece interact. Unit Tests depend on you coming with the inputs and defining the expected output. Doing this doesn't guarantee that it will work with the rest of your system.

Yes integration testing is a pain to deal with when you introduce code changes that deliberately changes the output. With our software we minimize by this by focusing on comparing the save result of a integration test with a saved correct result.

We have a tool that can use when we are sure that we are producing the correct results. It goes and loads up the old saved correct results and modifies them to work with the new setup.

Answer 6

First, 100% unit test coverage is not enough even at unit testing level: you cover only 100% of the instructions of your code. What about paths in your code? What about input or output domains?

Second, you don't know whether output from a sender unit is compatible with input from its receiver unit. This is the purpose of integration testing.

Finally, unit testing may be performed on a different environment than production. Integration testing may reveal discrepancies.

Answer 7

"opaque (when broken someone has to dive through all the layers to find out what is wrong)" -- this is exactly why integration tests are done - otherwise those opaque issues would show up in production environment.

Answer 8

I routinely see all sorts of issues uncovered by good integration testing - especially if you can automate some of your integration testing.

Unit tests are great, but you can accomplish 100% code coverage without 100% relevancy in your unit tests. You're really trying to test different things, right? In unit tests, you're looking for edge cases for a specific function, usually, whereas integration testing is going to show you problems at a higher level as all these functions interact.

If you build an API into your software, you can use this for automated integration testing - I've gotten a lot of mileage out of this in the past. I don't know that I'd go as far as to say that I'd dump unit tests in favor of integration tests, but when they're done right, they're a really powerful addition.

Answer 9

Definitions

I think it's important to define your terms before having this discussion.

Unit test tests a single unit in isolation. For me, that's a class. A unit test will create an object, invoke a method, and check a result. It answers the question "does my code do what I intended it to do?"

Integration test tests the combination of two components in the system. It is focused on the relationship between the components, not the components themselves. It answers the question "do these components work together as intended".

System test tests the whole software system. It answers the question "does this software work as intended?"

Acceptance test is an automated way for the customer answer the question "is this software what I think I want?". It is a kind of system test.

Note that none of these tests answer questions like "is this software useful?" or "is this software easy to use?".

All automated tests are limited by axiom "End-to-end is further than you think" - eventually a human has to sit down in front of a computer and look at your user interface.

Comparisons

Unit tests are faster and easier to write, faster to run, and easier to diagnose. They don't depend on "external" elements like a file system or a database, so they are much simpler/faster/reliable. Most unit tests continue to work as you refactor (and good unit tests are the only way to refactor safely). They absolutely require that your code be decoupled, which is hard, unless you write the test first. This combination of factors makes the Red/Green/Refactor sequence of TDD work so well.

System tests are hard to write, because they have to go through so much setup to get to a specific situation that you want to test. They are brittle, because any change in the behavior of the software before can affect the sequence leading up to the situation you want to test, even if that behavior isn't relevant to the test. They are dramatically slower than unit tests for similar reasons. Failures can be very difficult to diagnose, both because it can take a long time to get to the point of failure, and because so much software is involved in the failure. In some software, system tests are very difficult to automate.

Integration tests sit in between: they are easier to write, run, and diagnose than system tests, but with broader coverage than unit tests.

Recommendation

Use a combination of testing strategies to balance the costs and values of each.

Answer 10

This exact question was basically just asked a day ago. See this question for lots of the errors you could run into even with 100% code coverage.

Answer 11

You can only prove the presence of a bug using tests/coverage, but you can never prove that the code is bug-free using tests/coverage. This fact indicates the boundaries of testing/coverage. This is the same in mathematics, you can disprove a theorem by finding a counter example, but you can never prove a theorem by not finding a counter example. So testing and coverage are only a substitute for correctness proofs, which are so difficult to do that they are almost never used. Testing and coverage can improve quality of the code, but there is no guarantee. It remains a craft an not a science.