How do I get repeatable CPU-bound benchmark runtimes on Windows?

Question

We sometimes have to run some CPU-bound tests where we want to measure runtime. The tests last in the order of a minute. The problem is that from run to run the runtime varies by quite a lot (+/- 5%). We suspect that the variation is caused by activity from other applications/services on the system, eg:

• Applications doing housekeeping in their idle time (e.g. Visual Studio updating IntelliSense)
• Filesystem indexers
• etc..

What tips are there to make our benchmark timings more stable?

Currently we minimize all other applications, run the tests at "Above Normal" priority, and not touch the machine while it runs the test.

Answer 1

The usual approach is to perform lots of repetitions and then discard outliers. So, if the distractions such as the disk indexer only crops up once every hour or so, and you do 5 minutes runs repeated for 24 hours, you'll have plenty of results where nothing got in the way. It is a good idea to plot the probability density function to make sure you are understand what is going on. Also, if you are not interested in startup effects such as getting everything into the processor caches then make sure the experiment runs long enough to make them insignificant.

Answer 2

First of all, if it's just about benchmarking the application itself, you should use CPU time, not wallclock time as a measure. That's then (almost) free from influences of what the other processes or the system do. Secondly, as Dickon Reed pointed out, more repetitions increase confidence.

Answer 3

I do the following:

1. Call the method x times and measure the time
2. Do this n times and calculate the mean and standard deviation of those measurements

Try to get the x to a point where you're at a >1 second per measurement. This will reduce the noise a bit.

The mean will tell you the average performance of your test and the standard deviation the stability of your test/measurements.

I also set my application at a very high priority, and when I test a single-thread algorithm I associate it with one cpu core to make sure there is not scheduling overhead.

This code demonstrates how to do this in .NET:

Thread.CurrentThread.Priority = ThreadPriority.Highest;
Process.GetCurrentProcess().PriorityClass = ProcessPriorityClass.RealTime;
if (Environment.ProcessorCount > 1)
{
   Process.GetCurrentProcess().ProcessorAffinity = 
      new IntPtr(1 << (Environment.ProcessorCount - 1));
}

Answer 4

Quote from VC++ team blog, how they do performance tests:

To reduce noise on the benchmarking machines, we take several steps:

1. Stop as many services and processes as possible.
2. Disable network driver: this will turn off the interrupts from NIC caused by >broadcast packets.
3. Set the test’s processor affinity to run on one processor/core only.
4. Set the run to high priority which will decrease the number of context switches.
5. Run the test for several iterations.