What is Code Coverage and how do YOU measure it?

Question

What is Code Coverage and how do YOU measure it? I was asked this question regarding our automating testing code coverage. It seems to be that outside of automated tools it is more art than science. Anyone have any real world examples of how they use code coverage?

Answer 1

Code Coverage is simply a measure of the code that is tested. There are a variety of coverage criteria that can be measured, but typically it is the various paths, conditions, functions, and statements within a program that makeup the total coverage. The code coverage metric is the just a percentage of tests that execute each of these coverage criteria.

As far as how I go about tracking unit test coverage on my projects, I use static code analysis tools to keep track.

Answer 2

Code Coverage is a measurement of how many lines/blocks/arcs of your code are executed while the automated tests are running.

CC is collected by using a specialized tool to instrument the binaries to add tracing calls and run a full set of automated tests against the instrumented product. A good CC tools will give you not only the percentage of the code that is executed, but also will allow you to drill into the data and see exactly which lines of code were executed during particular test.

Our team uses Magellan - an in-house set of CC tools. If you are a .Net shop, VS has integrated tools to collect code coverage. You can also roll some custom tools, like this article describes.

If you are a C++ shop, Intel has some CC tools that runs for Windows and Linux, though I haven't used it. I've also heard there's the gcov tool for gcc, but I don't know anything about it and can't give you a link.

As to how we use it - code coverage is one of our exit criteria for each milestone. We have actually three CC numbers - CC from unit test (from dev team), CC from scenario test (from test team) and combined CC.

Btw, while CC is a good metric of how much testing you are doing, it is not necessarily a good metric of how well you are testing your product. There are other metrics you should use along with CC to ensure the quality.

Answer 3

Code coverage basically tests that how much of your code is covered under tests. So, if you have 90% code coverage than it means there is 10 code that is not covered under test. I know you might be thinking that 90% of the code is covered but you have to look from a different angle. What is stopping you to get 100% code coverage.

A good example will be this:

if(customer.IsOldCustomer()) {

}

else

{

}

Now, in the above code there are two paths/branches. If you are always hitting the "YES" branch then you are not covering the else part and it will be shown in the Code Coverage results. This is good because now you know that what is not covered and you can write a test to cover the else part. If there was no code coverage then you are just sitting on a time bomb to explode.

NCover is a good tool to measure code coverage.

Answer 4

Just remember, having "100% code-coverage" doesn't mean everything is tested completely - while it means every line of code is tested, it doesn't mean they are tested under every (common) situation..

I would use code-coverage to highlight bits of code that I should probably write tests for. For example, if whatever code-coverage tool shows myImportantFunction() isn't executed while running my current unit-tests, they should probably be improved.

Basically, 100% code-coverage doesn't mean your code is perfect. Use it as a guide to write more comprehensive (unit-)tests.

Answer 5

Code coverage has been explained well in the answers above. So this is more of an answer to the second part of the question.

We've used 3 tools to determine code-coverage.

1. JTest - a proprietary tool built over JUnit. (It generates unit tests as well)
2. Cobertura - an open source code coverage tool that can easily be coupled with JUnit tests to generate reports.

3. Emma - another - this one we've used for a slightly different purpose than unit testing. It has been used to generate coverage reports when the web-application is accessed by end-users. This coupled with web-testing tools (ex: Canoo) can give you very useful coverage reports which tell you how much code is covered during typical end user usage.

   We use these tools to

   1. . Review that developers have written good unit tests
   2. . Ensure that all code is traversed during black-box testing

Answer 6

For Perl there's the excellent Devel::Cover module which I regularly use on my modules.

If the build and installation is managed by Module::Build you can simply run ./Build testcover to get a nice HTML site that tells you the coverage per sub, line and condition, with nice colors making it easy to see which code path has not been covered.