Java Performance Testing

Question

I want to do some timing tests on a Java application.

This is what I am currently doing:

long startTime = System.currentTimeMillis();

doSomething();

long finishTime = System.currentTimeMillis();

System.out.println("That took: "+(finishTime-startTime)+ " ms");

Is there anything "wrong" with performance testing like this? What is a better way?

Duplicate: http://stackoverflow.com/questions/410437/is-stopwatch-benchmarking-acceptable

Answer 1

I'd imagine you'd want to doSomething() before you start timing too, so that the code is JITted and "warmed up".

HTH, Kent

Answer 2

Well that is just one part of performance testing. Depending on the thing you are testing you may have to look at heap size, thread count, network traffic or a whole host of other things. Otherwise I use that technique for simple things that I just want to see how long they take to run.

Answer 3

Keep in mind that the resolution of System.currentTimeMillis() varies between different operating systems. I believe Windows is around 15 msec. So if your doSomething() runs faster than the time resolution, you'll get a delta of 0. You could run doSomething() in a loop multiple times, but then the JVM may optimize it.

Answer 4

That's good when you are comparing one implementation to another or trying to find a slow part in your code (although it can be tedious). It's a really good technique to know and you'll probably use it more than any other, but be familiar with a profiling tool as well.

Answer 5

The one flaw in that approach is that the "real" time doSomething() takes to execute can vary wildly depending on what other programs are running on the system and what its load is. This makes the performance measurement somewhat imprecise.

One more accurate way of tracking the time it takes to execute code, assuming the code is single-threaded, is to look at the CPU time consumed by the thread during the call. You can do this with the JMX classes; in particular, with ThreadMXBean. You can retrieve an instance of ThreadMXBean from java.lang.management.ManagementFactory, and, if your platform supports it (most do), use the getCurrentThreadCpuTime method in place of System.currentTimeMillis to do a similar test. Bear in mind that getCurrentThreadCpuTime reports time in nanoseconds, not milliseconds.

This method isn't perfect, but is less subject to variations in system load.

Answer 6

Have you looked at the profiling tools in netbeans and eclipse. These tools give you a better handle on what is REALLY taking up all the time in your code. I have found problems that I did not realize by using these tools.

Answer 7

Japex may be useful to you, either as a way to quickly create benchmarks, or as a way to study benchmarking issues in Java through the source code.

Answer 8

The code shown in the question is not a good performance measuring code:

1. The compiler might choose to optimize your code by reordering statements. Yes, it can do that. That means your entire test might fail. It can even choose to inline the method under test and reorder the measuring statements into the now-inlined code.

2. The hotspot might choose to reorder your statements, inline code, cache results, delay execution...

3. Even assuming the compiler/hotspot didn't trick you, what you measure is "wall time". What you should be measuring is CPU time (unless you use OS resources and want to include these as well or you measure lock contestation in a multi-threaded environment).

The solution? Use a real profiler. There are plenty around, both free profilers and demos / time-locked trials of commercials strength ones.

Answer 9

Using a Java Profiler is the best option and it will give you all the insight that you need into the code. viz Response Times, Thread CallTraces, Memory Utilisations, etc

I will suggest you JENSOR, an open source Java Profiler, for its ease-of-use and no overheads on CPU. You can download it, instrument the code and will get all the info you need about your code.

You can download it from: http://jensor.sourceforge.net/