What type of errors could my code still contain even if I have 100% code coverage?

Question

Looking for concrete examples or links to concrete examples of such errors.

Thanks in advance.

Answer 1

Code coverage doesn't mean that your code is bug free in any way. It's an estimate on how well you're test cases cover your source code base. 100% code coverage would imply that every line of code is tested but every state of your program certainly is not. There's research being done in this area, I think it's referred to as finite state modeling but it's really a brute force way of trying to explore every state of a program.

A more elegant way of doing the same thing is something referred to as abstract interpretation. MSR (Microsoft Research) have released something called CodeContracts based on abstract interpretation. Check out Pex as well, they really emphasis cleaver methods of testing application run-time behavior.

I could write a really good test which would give me good coverage, but there's no guarantees that that test will explore all the states that my program might have. This is the problem of writing really good tests, which is hard.

Code coverage does not imply good tests

Answer 2

There can always be runtime exceptions: memory filling up, database or other connections not being closed etc...

Answer 3

Having 100% code coverage is not that great as one may think of it. Consider a trival example:

double Foo(double a, double b)
{
    return a / b;
}

Even a single unit test will raise code coverage of this method to 100%, but the said unit test will not tell us what code is working and what code is not. This might be a perfectly valid code, but without testing edge conditions (such as when b is 0.0) unit test is inconclusive at best.

Code coverage only tells us what was executed by our unit tests, not whether it was executed correctly. This is an important distinction to make. Just because a line of code is executed by a unit test, does not necessarily mean that that line of code is working as intended.

Listen to this for an interesting discussion.

Answer 4

Uh? Any kind of ordinary logic bug, I guess? Memory corruption, buffer overrun, plain old wrong code, assignment-instead-of-test, the list goes on. Coverage is only that, it lets you know that all code paths are executed, not that they are correct.

Answer 5

Well if your tests don't test the thing which happens in the code covered. If you have this method which adds a number to properties for example:

public void AddTo(int i)
{
NumberA += i;
NumberB -= i;
}

If your test only checks the NumberA property, but not NumberB, then you will have 100% coverage, the test passes, but NumberB will still contain an error.

Conclusion: a unit test with 100% will not guarantee that the code is bug-free.

Answer 6

Consider the following code:

int add(int a, int b)
{
  return a + b;
}

This code could fail to implement some necessary functionality (i.e. not meet an end-user requirements): "100% coverage" doesn't necessarily test/detect functionality which ought to be implemented but which isn't.

This code could work for some but not all input data ranges (e.g. when a and b are both very large).

Answer 7

1. "Data space" problems

Your (bad) code:

void f(int n, int increment)
{
  while(n < 500)
  {
    cout << n;
    n += increment;
  }
}

Your test:

f(200,100);

Bug in real world use:

f(200,0);

My point: Your test may cover 100% of the lines of your code but it will not (typically) cover all your possible input data space, i.e. the set of all possible values of inputs.

2. Testing against your own mistake

Another classical example is when you just take a bad decision in design, and test your code against your own bad decision.

E.g. The specs document says "print all prime numbers up to n" and you print all prime numbers up to n but excluding n. And your unit tests test your wrong idea.

3. Undefined behaviour

Use the value of uninitialized variables, cause an invalid memory access, etc. and your code has undefined behaviour (in C++ or any other language that contemplates "undefined behaviour"). Sometimes it will pass your tests, but it will crash in the real world.

...

Answer 8

Errors in tests :)

Answer 9

Code coverage doesn't mean anything, if your tests contain bugs, or you are testing the wrong thing.

As a related tangent; I'd like to remind to you that I can trivially construct an O(1) method that satisfies the following pseudo-code test:

sorted = sort(2,1,6,4,3,1,6,2);

for element in sorted {
  if (is_defined(previousElement)) {
    assert(element >= previousElement);
  }

  previousElement = element;
}

bonus karma to Jon Skeet, who pointed out the loophole I was thinking about

Answer 10

As I haven't seen it mentioned yet, I'd like to add this thread that code coverage does not tell you what part of your code is bugfree.

It only tells you what parts of your code is guaranteed to be untested.

Answer 11

Code coverage usually only tells you how many of the branches within a function are covered. It doesn't usually report the various paths that could be taken between function calls. Many errors in programs happen because the handoff from one method to another is wrong, not because the methods themselves contain errors. All bugs of this form could still exist in 100% code coverage.

Answer 12

In a recent IEEE Software paper "Two Mistakes and Error-Free Software: A Confession", Robert Glass argued that in the "real world" there are more bugs caused by what he calls missing logic or combinatorics (which can't be guarded against with code coverage tools) than by logic errors (which can).

In other words, even with 100% code coverage you still run the risk of encountering these kinds of errors. And the best thing you can do is--you guessed it--do more code reviews.

The reference to the paper is here and I found a rough summary here.

Answer 13

Argument validation, aka. Null Checks. If you take any external inputs and pass them into functions but never check if they are valid/null, then you can achieve 100% coverage, but you will still get a NullReferenceException if you somehow pass null into the function because that's what your database gives you.

also, arithmetic overflow, like

int result = int.MAXVALUE + int.MAXVALUE;

Code Coverage only covers existing code, it will not be able to point out where you should add more code.

Answer 14

works on my machine

Many things work well on local machine and we cannot assure that to work on Staging/Production. Code Coverage may not cover this.

Answer 15

I don't know about anyone else, but we don't get anywhere near 100% coverage. None of our "This should never happen" CATCHes get exercised in our tests (well, sometimes they do, but then the code gets fixed so they don't any more!). I'm afraid I don't worry that there might be a Syntax/Logic error in a never-happen-CATCH

Answer 16

Your product might be technically correct, but not fulfil the needs of the customer.

Answer 17

FYI, Microsoft Pex attempts to help out by exploring your code and finding "edge" cases, like divide by zero, overflow, etc.

This tool is part of VS2010, though you can install a tech preview version in VS2008. It's pretty remarkable that the tool finds the stuff it finds, though, IME, it's still not going to get you all the way to "bulletproof".

Answer 18

Perform a 100% code coverage, i.e., 100% instructions, 100% input and output domains, 100% paths, 100% whatever you think of, and you still may have bugs in your code: missing features.

Answer 19

Code Coverage doesn't mean much. What matters is whether all (or most) of the argument values that affect the behavior are covered.

For eg consider a typical compareTo method (in java, but applies in most languages):

//Return Negative, 0 or positive depending on x is <, = or > y
int compareTo(int x, int y) {
   return x-y;
}

As long as you have a test for compareTo(0,0), you get code coverage. However, you need at least 3 testcases here (for the 3 outcomes). Still it is not bug free. It also pays to add tests to cover exceptional/error conditions. In the above case, If you try compareTo(10, Integer.MAX_INT), it is going to fail.

Bottomline: Try to partition your input to disjoint sets based on behavior, have a test for at least one input from each set. This will add more coverage in true sense.

Also check for tools like QuickCheck (If available for your language).

Answer 20

As mentioned in many of the answers here, you could have 100% code coverage and still have bugs.

On top of that, you can have 0 bugs but the logic in your code may be incorrect (not matching the requirements). Code coverage, or being 100% bug-free can't help you with that at all.

A typical corporate software development practice could be as follows:

1. Have a clearly written functional specification
2. Have a test plan that's written against (1) and have it peer reviewed
3. Have test cases written against (2) and have them peer reviewed
4. Write code against the functional specification and have it peer reviewed
5. Test your code against the test cases
6. Do code coverage analysis and write more test cases to achieve good coverage.

Note that I said "good" and not "100%". 100% coverage may not always be feasible to achieve -- in which case your energy is best spent on achieving correctness of code, rather than the coverage of some obscure branches. Different sorts of things can go wrong in any of the steps 1 through 5 above: wrong idea, wrong specification, wrong tests, wrong code, wrong test execution... The bottom line is, step 6 alone is not the most important step in the process.

Concrete example of wrong code that doesn't have any bugs and has 100% coverage:

/**
 * Returns the duration in milliseconds
 */
int getDuration() {
    return end - start;
}

// test:

start = 0;
end = 1;
assertEquals(1, getDuration()); // yay!

// but the requirement was:
// The interface should have a method for returning the duration in *seconds*.

Answer 21

Almost anything.

Have you read Code Complete? (Because StackOverflow says you really should.) In Chapter 22 it says "100% statement coverage is a good start, but it is hardly sufficient". The rest of the chapter explains how to determine which additional tests to add. Here's a brief taster.

• Structured basis testing and data flow testing means testing each logic path through the program. There are four paths through the contrived code below, depending on the values of A and B. 100% statement coverage could be achieved by testing only two of the four paths, perhaps f=1:g=1/f and f=0:g=f+1. But f=0:g=1/f will give a divide by zero error. You have to consider if statements and while and for loops (the loop body might never be executed) and every branch of a select or switch statement.

  If A Then
f = 1
Else
f = 0
End If
If B Then
g = f + 1
Else
g = f / 0
End If

• Error guessing - informed guesses about types of input that often cause errors. For instance boundary conditions (off by one errors), invalid data, very large values, very small values, zeros, nulls, empty collections.

And even so there can be errors in your requirements, errors in your tests, etc - as others have mentioned.