How can I measure CppUnit test coverage (on win32 and Unix)?

Question

I have a very large code base that contains extensive unit tests (using CppUnit). I need to work out what percentage of the code is exercised by these tests, and (ideally) generate some sort of report that tells me on a per-library or per-file basis, how much of the code was exercised.

Here's the kicker: this has to run completely unnatended (eventually inside a continuous integration build), and has to be cross platform (well, WIN32 and *nix at least).

Can anyone suggest a tool, or set of tools that can help me do this? I can't change away from CppUnit (nor would I want to - it kicks ass), but otherwise I'm eager to hear any recommendations you might have.

Cheers,

Answer 1

Which tool should I use?

This article describes another developers frustrations searching for C++ code coverage tools. The author's final solution was Bullseye Coverage.

Bullseye Coverage features:

• Cross Platform Support (win32, unix, and embedded), (supports linux gcc compilers and MSVC6)
• Easy to use (up and running in a few hours).
• Provides "best" metrics: Function Coverage and Condition/Decision Coverage.
• Uses source code instrumentation.

As for hooking into your continuous integration, it depends on which CI solution you use, but you can likely hook the instrumentation / coverage measurement steps into the make file you use for automated testing.

---

Testing Linux vs Windows?

So long as all your tests run correctly in both environments, you should be fine measuring coverage on one or the other. (Though Bullseye appears to support both platforms). But why aren't you doing continuous integration builds in both environments?? If you deliver to clients in both environments then you need to be testing in both.

For that reason, it sounds like you might need to have two continuous build servers set up, one for a linux build and one for a windows build. Perhaps this can be easily accomplished with some virtualization software like vmware or virtualbox. You may not need to run code coverage metrics on both OSs, but you should definitely be running your unit tests on both.

Answer 2

If you can use GNU GCC as your complier, then the gcov tool works well. It's very easy to fully automate the whole process.

Answer 3

I guess I should have specified the compiler - we're using gcc for Linux, and MSVC 6 (yeah I know, it's old, but it works (mostly) for us) for WIn32.

For that reasons, gcov won't work for our Win32 builds, and Bullseye won't work for our Linux builds.

Then again maybe I only need coverage in one OS...

Answer 4

If you are using the GCC toolchain, gcov is going to get you source, functional, and branch coverage statistics. gcov works fine for MinGW and Cygwin. This will allow you to get coverage statistics as well as emitting instrumented source code that allows you to visualize unexecuted code.

However, if you really want to hit it out of the park with pretty reports, using gcov in conjunction with lcov is the way to go. lcov will give you bar reports scoped to files and directories, functional coverage statistics, and color coded source file browsing to show coverage (green means executed, red means not...).

lcov is easy on Linux, but may require some perl hacking on Cygwin. I personally have had some problems executing the scripts (lcov is implemented in perl) on Windows. I've gotten a hacked up version to work, but be forewarned.

Another approach is doing the gcov emit on windows, and doing the lcov post processing on Linux, where it will surely work out of the box.

Answer 5

i have sort of the same problem... i use Bullseye for code coverage for windows, but i am having difficulty running it on a linux platform... can anyone please give me some instructions to make work in linux?

Answer 6

Check out our SD C++ Test Coverage tool. It can be obtained for GCC, and for MSVC6.

It has low overhead probe data collection, a nice display of coverage data overlayed on your code, and complete report generation with rollups on coverage across the method/class/file/directory levels.