How do I write a correct micro-benchmark in Java?

Question

As the title says. How do you write (and run) a correct micro-benchmark in Java?

I'm looking here for code samples and comments illustrating various things to think about.

Example: Should the benchmark measure time/iteration or iterations/time, and why?

Near Duplicate of: http://stackoverflow.com/questions/410437/is-stopwatch-benchmarking-acceptable

Answer 1

See this question from a few minutes ago for some related info.

edit: sorry, this is not supposed to be a answer. I should've posted as a comment.

Answer 2

Important things for Java benchmarks are:

• Warm up the JIT first by running the code several times before timing it
• Make sure you run it for long enough to be able to measure the results in seconds or (better) tens of seconds
• While you can't call System.gc() between iterations, it's a good idea to run it between tests, so that each test gets a "clean" memory space to work with
• I like to display iterations and time, and a score of time/iteration which can be scaled such that the "best" algorithm gets a score of 1.0 and others are scored in a relative fashion. This means you can run all algorithms for a longish time, varying both number of iterations and time, but still getting comparable results.

I'm just in the process of blogging about the design of a benchmarking framework in .NET. I've got a couple of earlier posts which may be able to give you some ideas - not everything will be appropriate, of course, but some of it may be.

Answer 3

There are many possible pitfalls for writing micro-benchmarks in Java.

First: You have to calculate with all sorts of events that take time more or less random: Garbage collection, caching effects (of OS for files and of CPU for memory), IO etc.

Second: You cannot trust the accuracy of the measured times for very short intervals.

Third: The JVM optimizes your code while executing. So different runs in the same JVM-instance will become faster and faster.

My recommendations: Make your benchmark run some seconds, that is more reliable than a runtime over milliseconds. Warm up the JVM (means running the benchmark at least once without measuring, that the JVM can run optimizations). And run your benchmark multiple times (maybe 5 times) and take the median-value. Run every micro-benchmark in a new JVM-instance (call for every benchmark new Java) otherwise optimization effects of the JVM can influence later running tests. Don't execute things, that aren't executed in the warmup-phase (as this could trigger class-load and recompilation).

Answer 4

If you are trying to compare two algorithms, do at least two benchmarks on each, alternating the order. i.e.:

for(i=1..n)
  alg1();
for(i=1..n)
  alg2();
for(i=1..n)
  alg2();
for(i=1..n)
  alg1();

I have found some noticeable differences (5-10% sometimes) in the runtime of the same algorithm in different passes..

Also, make sure that n is very large, so that the runtime of each loop is at the very least 10 seconds or so. The more iterations, the more significant figures in your benchmark time and the more reliable that data is.

Answer 5

Make sure you somehow use results which are computed in benchmarked code. Otherwise your code can be optimized away.

Answer 6

Should the benchmark measure time/iteration or iterations/time, and why?

It depends on what you are trying to test. If you are interested in latency, use time/iteration and if you are interested in throughput use iterations/time.

Answer 7

Here are some microbenchmarking frameworks:

http://japex.dev.java.net/

http://ellipticgroup.com/html/benchmarkingArticle.html

Answer 8

The Java Specialists have had a number of articles on performance. They have good examples.

Answer 9

Tips about writing micro benchmarks from the creators of Java HotSpot:

Rule 0: Read a reputable paper on JVMs and micro-benchmarking. A good one is Brian Goetz, 2005. Do not expect too much from micro-benchmarks; they measure only a limited range of JVM performance characteristics.

Rule 1: Always include a warmup phase which runs your test kernel all the way through, enough to trigger all initializations and compilations before timing phase(s). (Fewer iterations is OK on the warmup phase. The rule of thumb is several tens of thousands of inner loop iterations.)

Rule 2: Always run with -XX:+PrintCompilation, -verbose:gc, etc., so you can verify that the compiler and other parts of the JVM are not doing unexpected work during your timing phase.

Rule 2.1: Print messages at the beginning and end of timing and warmup phases, so you can verify that there is no output from Rule 2 during the timing phase.

Rule 3: Be aware of the difference between -client and -server, and OSR and regular compilations. The -XX:+PrintCompilation flag reports OSR compilations with an at-sign to denote the non-initial entry point, for example: Trouble$1::run @ 2 (41 bytes). Prefer server to client, and regular to OSR, if you are after best performance.

Rule 4: Be aware of initialization effects. Do not print for the first time during your timing phase, since printing loads and initializes classes. Do not load new classes outside of the warmup phase (or final reporting phase), unless you are testing class loading specifically (and in that case load only the test classes). Rule 2 is your first line of defense against such effects.

Rule 5: Be aware of deoptimization and recompilation effects. Do not take any code path for the first time in the timing phase, because the compiler may junk and recompile the code, based on an earlier optimistic assumption that the path was not going to be used at all. Rule 2 is your first line of defense against such effects.

Rule 6: Use appropriate tools to read the compiler's mind, and expect to be surprised by the code it produces. Inspect the code yourself before forming theories about what makes something faster or slower.

Rule 7: Reduce noise in your measurements. Run your benchmark on a quiet machine, and run it several times, discarding outliers. Use -Xbatch to serialize the compiler with the application, and consider setting -XX:CICompilerCount=1 to prevent the compiler from running in parallel with itself.

Answer 10

Some months ago there was a two-part series on IBM developerWorks on micro-benchmarks in Java, with code. I'm not an expert in VM internals but these articles seem to go to a great depth of detail, considering - and explaining - various types of VM optimizations, JIT time, GC, timer resolution, and also a somewhat in depth discussion of statistics.

Part 1

Part 2

Edit : actually this is the same material as a link in another anwser (http://www.ellipticgroup.com/html/benchmarkingArticle.html). So, hmm, these are just my 2 cents on this framework ;)

Answer 11

John Nilsson,

I've seen on your reply http://stackoverflow.com/questions/181892/problem-reading-from-a-device-with-libusb/188140#188140 that you have an advanced knowledge about libusb.

I'm a brazilian student of Computer Science. I'm developing an external usb gadget, and need to develop a device driver to operate it. It's like 3 LEDs flashing through an interface in the screen.

Could you help me to develop the device driver using libusb for windows? (libusb-win32) As it's ported to windows, it have the same functions than the linux version, right? So, as described in the project page, we only have to pay attention to usb_busses function, using usb_get_busses instead.

Please, could you help me? My teachers are expecting this driver, but I really have no idea on how start developing it. I already have installed libsub-win32, XP DDK and Visual C++. The device isn't built yet, is there any form of emulating it?

Thanks!! reply in: service_stack at hlera.com