Java: measure code coverage for remote scripting tests

Question

We have an application which is deployed on JBoss 5.1, JDK 1.6. We also have scripts written in PowerShell for testing. These scripts access the application using a web-service. I would like to check the code coverage of the scripts. Any ideas? Most of the tools I saw are checking a JUnit test coverage and I don't see how we can use them.

Answer 1

I use emma coverage tool integrated with unit testing project build phase, however, tool's documentation says that it's fairly simple to get code coverage at situation you described.

Answer 2

Code cover is a great tool. For your case you should use command-line interface, that might you incorporate with existing PowerShell scripts.

Answer 3

AFAIK, all code coverage tools use the same concept (I'll omit the reporting and checking part):

1. First instrument the code (i.e. place markers).
2. Then run tests to execute the instrumented code (to activate markers and collect data).

For the second step, the common use case is indeed to run JUnit tests but your tests don't have to be JUnit tests. Actually, they don't even have to be automated.

And the instrumented code doesn't have to be executed in the context of a unit test, it can be packaged in a WAR/EAR and deployed on a container (this will just require a bit more work).

For Cobertura, this is what we can read in the Frequently Asked Questions:

Using Cobertura with a Web Application

I have automated tests that use HttpUnit/HtmlUnit/Empirix/Rational Robot, can I use Cobertura?

Yes! The process is a bit more involved, but the concept is the same. First instrument your compiled classes. Then create your war file. Then deploy the war file into your application server (Tomcat, JBoss, WebLogic, WebSphere, etc). Now run your tests.

As your classes are accessed, they will create a "cobertura.ser" file on the disk. You may need to dig around a bit to find it. Cobertura puts this file in what it considers to be the current working directory. Typically this is the directory that the application server was started from (for example, C:\Tomcat\bin) Note: This file is not written to the disk until the application server exits. See below for how to work around this.

Now that you know where the cobertura.ser file is, you should modify your deploy step so that it moves the original cobertura.ser to the appropriate directory in your application server, and then moves it back when finished testing. Then run cobertura-report.

[...]

For Emma, this is what the documentation says:

3.11. How do I use EMMA in {WebLogic, Websphere, Tomcat, JBoss, ...}?

First of all, there is little chance that you will be able to use the on-the-fly mode (emmarun) with a full-blown J2EE container. The reason lies in the fact that many J2EE features require specialized classloading that will happen outside of EMMA instrumenting classloader. The server might run fine, but you will likely get no coverage data.

Thus, the correct procedure is to instrument your classes prior to deployment (offline mode). Offline instrumentation always follows the same compile/instrument/package/deploy/get coverage/generate reports sequence. Follow these steps:

1. use EMMA's instr tool to instrument the desired classes. This can be done as a post-compilation step, before packaging. However, many users also find it convenient to let EMMA process their jars directly (either in-place, using overwrite mode, or by creating separate instrumented copies of everything, in fullcopy mode);
2. do your J2EE packaging as normal, but do not include emma.jar as a lib at this level, that is, within your .war, .ear, etc;
3. locate whichever JRE is used by the container and copy emma.jar into its /lib/ext directory. If that is impossible, add emma.jar to the server classpath (in a server-specific way);
4. deploy your instrumented classes, .jars, .wars, .ears, etc and exercise/test your J2EE application via your client-side testcases or interactively or whichever way you do it;
5. to get a coverage dump file, you have three options described in What options exist to control when EMMA dumps runtime coverage data?. It is highly recommended that you use coverage.get control command with the ctl tool available in v2.1.

For clover, check the Working with Distributed Applications page.