Best way to measure execution time in automated regression tests

Question

I have some code that I want to measure the speed of while it runs continuously in automated regression tests. The purpose of this would be to alert me to changes made to the code which have had a negative impact on performance.

In pseudo-code, I want something like this:

cpuTimer.start
runTest
cpuTimer.stop

diff = cpuTimer.getDuration

if diff > prevDiff // Perhaps to within a tolerance
    failTest

I'm looking at ThreadMXBean#getCurrentThreadCpuTime() for this, but the key problem is that the automated tests will be run on a wide range of different developer's pcs, and will be automatically farmed out to test servers with a range of different hardware and capabilities.

Will this work, or will the numbers go awry?

How should this problem be solved? Is there a better way? Is there a standard tool for this sort of caper?

Answer 1

One approach would be to analyze the standard xml files that are output from the junit running in both maven and ant. These contain all the data you need. Although there are tools to generate reports based on these, I am not aware of any aggregators/watchers.

If you put these in a database table with fields hostname, testname, executiontime and buildNumber you should be able to do most of the things you'd need.

Answer 2

You might try to look at Perf4J. I haven't used it yet, but it's on my list of things to investigate.

From their homepage:

Perf4J is to System.currentTimeMillis() as log4j is to System.out.println()

The Developer Guide is a nice introduction.