Determining which tests cover a line of code

Question

Is there a way to determine the set of unit tests that will potentially execute a given line of code? In other words, can you automatically determine not just whether a given line is covered, but the actual set of tests that cover it?

Consider a big code base with, say, 50K unit tests. Clearly, it could take a LONG time to run them all--hours, if not days. Working in such a code base, you'd like to be able to execute some subset of all the unit tests, including only those that cover the line (or lines) that you just touched. Sure, you could find some manually and run those, but I'm looking for a way to do it faster, and more comprehensively.

If I'm thinking about this correctly, it should be possible. A tool could statically traverse all the code paths leading out of each unit test, and come up with a slice of the program reachable from that test. And you should then (theoretically) be able to compute the set of unit tests that include a given line in their slice, meaning that the line could be executed by that test ("could" rather than "will" because the actual code path will only be determined at run time based on the inputs or other conditions). A given line of code could have a massive number of tests that execute it (say, code in a shared library), whereas other lines might have few (or no) tests covering them.

So:

1. Is my reasoning sound on this idea? Could it theoretically be done, or is there something I'm leaving out?

2. Is there already a tool out there that can do this? Or, is this a common thing with a name I haven't run into? Pointers to tools in the java world, or to general research on the subject, would be appreciated.

Answer 1

I'm pretty sure Clover will show you which tests validate each line of code. So you could manually execute the tests by looking at the coverage reports. They also have a new API which you might be able to use to write an IDE plugin that would let you execute the tests that cover a line of code.

Answer 2

The following presentation discusses how to compute the program slice executed by a unit test. It answers the question of, "can you determine the test coverage without executing the program?" and basically sketches the idea you described... with the additional bit of work to actually implement it.

You might note that computing a program slice isn't a computationally cheap task. I'd guess that computing a slice (a symbolic computation) is generally slower than executing a unit test, so I'm not sure that you'll save any time doing this. And a slice is a conservative approximation of the affected part of the program, so the answer you get back will include program parts that actually don't get executed.

You might be better off instead to run all those 50,000 unit tests once, and collect the coverage data for each one. In this case, when some code fragment is updated, it is possible to determine statically whether the code a particular test executes includes the code you changed or not, and thus you can identify tests that have to be executed again. You can skip executing the rest of the tests.

My company builds a family of test coverage tools. Our next release of these tools will have this kind of incremental regression testing capability.