The value of test code coverage tools..

Question

We've started using Part Cover to track test code coverage of our application. IMO its a great tool for getting an overall score for your tests and for highlighting test areas where you might have been a bit lazy with tests, but today I wrote a test and realised that it didn't really test anything useful, it just increased my coverage!

If you are TDD, then you only write code to pass a test, and the tests are richly describing all the functionality required by the application. So in this scenario is it still very valuable to have coverage analysis?

For those of you that have coverage tools, how religiously do you adhere to keeping the coverage at 100% and do you ever find yourself writing tests that don't really test anything, but just to keep your coverage up? Isn't this a bad thing ?

Answer 1

Coverage tools should only be used to tell you what has not been tested. The scenario you pointed out illustrates why you can't rely on them to show you what code has been tested. Writing tests just so the coverage is 100% is pointless (as you suspected), and it's so easy to game that this isn't really a useful metric. I used to try and stay at or near 100%, but I came to the same conclusion that you did. I was writing tests that didn't really test anything just so the numbers were right. Use the tools to spot areas that you haven't tested yet, then write good tests or accept the fact that those parts of the code aren't critical.

Answer 2

Good rationalisation ;) But we are human after all, and I for one sleep much better at night knowing that an untested method or path hasn't made it into production.

Answer 3

If you're doing pure TDD, there's less value to code coverage because as you say, you only write code from tests so you should be at around 100% anyway. but then, it's probably pretty rare (and at times not possible) to be doing it so purely.

if you aren't doing pure TDD, 100% is a pretty unrealistic target anyway. I usually try to go for Roy Osherove's method and only test things with logic (e.g. not straight getters/setter or pass-throughs). But then, higher is always better, and it can be tempting to put a couple more tests in there to increase that coverage..!

Answer 4

I'll play devil's advocate: if increasing your coverage meant writing a test that "didn't test anything useful," then why was that code there? To me, this would be an argument to remove some mainline code.

Or to develop a test that does do something useful. For example, you may consider that it's not useful to test setters and getters. Neither do I. However, those methods should be tested while testing something else. Otherwise, again, why are they there?

But you raise a good point that coverage tools should not be an end in themselves. Especially since they can't tell you what code you need to write.

I've gone into more detail here: http://www.kdgregory.com/index.php?page=junit.coverage