Java code coverage in Hudson

Question

I'm migrating a couple of projects from an ant build to a maven one. The build server is , and will remain, Hudson.

I've been having trouble recording code coverage in hudson with cobertura due to the tests run and recorded twice problem.

The project is multi-module and it would be nice, although not required, to have an aggregated output of the code coverage data.

All in all, the solution I'm looking for must:

• run automated tests for all modules and record the results once;
• display the individual module code coverage in Hudson;
• be easily configured once for the whole project, not in every module.

The solution can be based on Cobertura, or Emma, or any other java code coverage tool.

---

Update: Running the tests with Emma still duplicates the results and there's no merge capability, so it's not really usable with multi-module builds.

Answer 1

Robert,

I had this problem as well and found that Hudson does not double report if you setup the project as a freestyle project rather than a Maven2 project. You do lose some of the niceness of having a maven2 project, but for us, it was a trade we had to make.

Jeff

Answer 2

We use free style projects and don't have this problem, so as indicated, this may be the source of your problem.

To provide the merging features, we created our own artifact repository (we aren't using Maven). At the end of each build, we copy the cobertura.ser file to a network share, renaming it in the process. We have a consolidated view job that copies all of the cobertura files and the source code files (another build artifact copied to the network share) into the local build directory and generate the Cobertura report.

The lack of a standard artifact repository within Hudson is a bit frustrating, but it makes sense give the authors typically use Maven for those needs. Our build process runs over multiple servers so we can't just use relative paths into the other job directories.

Note, we do the same thing for other metrics: test results, JavaNCSS, ect. and joined either using the correct tools or some custom code.

We use the same repository for traditional build artifacts: DLLs, JARs, installation scripts.

Answer 3

Have you considered Atlassian's Clover?

The maven-clover2-plugin has a new goal: clover2:setup which will simply instrument your tests without forking the lifecycle, or running the tests twice.

You would define the goals to run in Hudson like so:

mvn clover2:setup verify clover2:aggregate clover2:clover

The maven-clover2-plugin is absolutely free to try for 30 days.

Cheers, Nick - Atlassian Clover.

Answer 4

It's a bit hackish, but the approach I'm using is to use a modified version of the Maven cobertura plugin (which is available from their repo). It provides a cobertura:generate-report target, so that you can insert cobertura:instrument and cobertura:generate-report into your lifecycle before and after tests run, respectively. That'll get you the coverage data you want without the duplicate test execution/recording.

The underlying problem is that all the non-clover Maven coverage plugins I've run into are built around the idea of running the tests with coverage separately from the main test execution in the Maven lifecycle. This, obviously, results in two sets of test executions. If you're using a freestyle project, you'll only get one set of tests recorded (since, even with two test executions, there's only one copy of the test output), but the Maven project type actually intercepts the Maven mojo executions and records test output/results at test execution time, rather than all at once at the end of the build as freestyle projects do. This has plenty of advantages, but it also has the rather glaring disadvantage that a single test getting executed twice gets counted as two tests.

That said, while I've seen strong arguments for running tests against both non-instrumented and instrumented code, I prefer to only run the tests once, against instrumented code - not just because of the Maven/Hudson issues, but because when you've got tests that take 45 minutes, it seems quite frankly silly to run them twice to generate the same result.

Answer 5

See SD Java Test Coverage for an extremely low overhead tool with a nice GUI. I'm not sure I understand your "run twice" problem, but if you ran (the same deterministic) tests twice with the SD tools, you'll get the the same test coverage data, e.g., its idempotent. If your tests are nondeterministic, you'll get two different test runs, but these tools easily merges the results of several runs into a single overall summary.

They also handle extremely large applications, and handle multiple threaded applications pretty well (small timing splinters can make the answer slightly inaccurate in theory, but practice this simply isn't an issue).

Answer 6

Sonar is a very cool tool that is easily integrated with Hudson, I really like its organization with multi-module projects. You should give it a try