Testing the performance of a C++ app

Question

I'm trying to find a way to test how long it takes a block of C++ code to run. I'm using it to compare the code with different algorithms and under different languages, so ideally I would like a time in seconds / milliseconds. In Java I'm using something like this:

long startTime = System.currentTimeMillis();

function();

long stopTime = System.currentTimeMillis();
long elapsedTime = stopTime - startTime;

Is there a good way to get an accurate time like that in C++ (Or should I use some other means of benchmarking)?

edit: Thanks Andy, that was what I was looking for.

Answer 1

Execute the function a few thousand times to get an accurate measurement.

A single measurement might be dominated by OS events or other random noise.

Answer 2

You can use the time() function to get a timer with a resolution of one second. If you need more resolution, you could use gettimeofday(). The resolution of that depends on your operating system and runtime library.

Answer 3

In Windows, you can use high performance counters to get more accurate results:

You can use the QueryPerformanceFrequency() function to get the number of high frequency ticks per seconde and the user the QueryPerformanceCounter() before and after the function you want to time.

Of course, this method is not portable...

Answer 4

Have you considered actually using a profiler? Visual Studio Team System has one built in, but there are others available like VTune and GlowCode.

See also http://stackoverflow.com/questions/67554/whats-the-best-free-c-profiler-for-windows-if-there-are

Answer 5

You can simply use time() within your code to measure within accuracy of seconds. Thorough benchmarks should run many iterations for accuracy so seconds should be a large enough margin. If you are using linux, you can use the time utility as provided by the command line like so:

[john@awesome]$time ./loops.sh

real    0m3.026s
user    0m4.000s
sys     0m0.020s

Answer 6

On unix systems (Linux, Mac, etc.) you can use time utility like so:

$ time ./my_app

Answer 7

If your function is very fast, a good practice is to time the function in a loop and then subtract the loop overhead.

Something like this:

int i;
int limit=1000000;
int t0=getTime();
for(i=0; i < limit; ++i)
   ;
int t1=getTime();
int loopoverhead = t1-t0;
t0=getTime();
for(i=0; i < limit; ++i)
    function();
t1=getTime();
double tfunction = (1.0/limit)*((t1-t0)-loopoverhead);

Answer 8

I always use boost::timer or boost::progress_timer.

psudo-code:

#include <boost/timer.hpp>

boost::timer t;

func1();
cout << "fun1: " << t.elapsed();

t.restart();
func2();
cout << "func2(): " << t.elapsed();

Answer 9

Use the best counter available on your platform, fall back to time() for portability. I am using QueryPerformanceCounter, but see the comments in the other reply.

General advise:

The inner loop should run at least about 20 times the resolution of your clock, to make the resolution error < 5%. (so, when using time() your inner loop should run at least 20 seconds)

Repeat these measurements, to see if they are consistent.

I use an additional outer loop, running ten times, and ignoring the fastest and the slowest measurement for calculating average and deviation. Deviation comes handy when comparing two implementations: if you have one algorithm taking 2.0ms +/-.5, and the other 2.2 +/- .5, the difference is not significant to call one of them "faster". (max and min should still be displayed). So IMHO a valid performance measurement should look something like this:

10000 x 2.0 +/- 0.2 ms (min = 1.2, , max=12.6), 10 repetitions

If you know what you are doing, purging the cache and setting thread affinity can make your measurements much more robust.

However, this is not without pifalls. The more "stable" the measurement is, the less realistic it is as well. Any implementation will vary strongly with time, depending on the state of data and instruction cache. I'm lazy here, useing the max= value to judge first run penalty, this might not be sufficient for some scenarios.

Answer 10

If you want to check you performance, you should consider measuring used processor time, not the real time you are trying to measure now. Otherwise you might get quite inaccurate times if some other application running in background decides to do some heavy calculations at the same time. Functions you want would be GetProcessTimes on Windows and getrusage on Linux.

Also you should consider using profilers, as other people suggested.

Answer 11

Run it 1000 times as 100 iterations * 10 iterations, where you unroll the inner loop to minimize overhead. Then seconds translate to milliseconds.

As others have pointed out, this is a good way to measure how long it takes.

However, if you also want to make it take less time, that is a different goal, and needs a different technique. My favorite is this.

Answer 12

What's wrong with clock() and CLOCKS_PER_SEC? They are standard C89.

Something like (nicked from MSDN):

   long i = 6000000L;
   clock_t start, finish;
   double  duration;

   // Measure the duration of an event.
   printf( "Time to do %ld empty loops is ", i );
   start = clock();
   while( i-- ) 
      ;
   finish = clock();
   duration = (double)(finish - start) / CLOCKS_PER_SEC;
   printf( "%2.1f seconds\n", duration );

Answer 13

I'm going to go ahead and answer my own question by saying that the link (http://stackoverflow.com/questions/275004/c-timer-function-to-provide-time-in-nano-seconds) in the comment posted by Andy White was what I was looking for.