I'm looking for a tool that can give me more meaningful metrics about code coverage for my team. For instance, two things I'd like to see:
Has anyone done anything like this before? What tools are available? Specifically, I'm working in Java and am interested in either free or commercial solutions.
You have two problems:
1) What's the coverage of the code that I presently have? 2) What's the coverage compared to what I used to have?
I think you need three tools:
a) to compute coverage of your application this instant,
b) something that computes the difference between what you have now, and what you had the last time you computed code coverage,
c) something to pick out the coverage data that changes.
You can get a) from a number of Java test coverage tools, but you want it in a form that you can process for step c). The SD Java Test Coverage Tool does provide that data in a usable form. The test coverage is reported in XML with line number data so you can easily build a tool to extract what you want.
b) Any diff tool will do to identify file differences under the assumptionst that the files from the previous version and from this version line up one for one. Where they don't, you can simply assume the test coverage data is completely different :-} However, you want the smallest diff you can get; see the SD Java Smart Differencer Tool for a tool that often computes much smaller deltas than conventional diff. The output of the Smart Differencer is machine readable.
c) Don't have an off-the-shelf answer for this, but a Perl script that combined the information from a) and b) would likely do the trick. What you want to report is, for each explicit delta identified by the diff tool, what's the coverage data, and for all code not changed according to the diff tool, whether the new coverage is less than the old coverage.