When benchmarking, what causes a lag between CPU time and "elapsed real time"?

Question

I'm using a built-in benchmarking module for some quick and dirty tests. It gives me:

• CPU time
• system CPU time (actually I never get any result for this with the code I'm running)
• the sum of the user and system CPU times (always the same as the CPU time in my case)
• the elapsed real time

I didn't even know I needed all that information.

I just want to compare two pieces of code and see which one takes longer. I know that one piece of code probably does more garbage collection than the other but I'm not sure how much of an impact it's going to have.

Any ideas which metric I should be looking at?

And, most importantly, could someone explain why the "elapsed real time" is always longer than the CPU time - what causes the lag between the two?

Answer 1

Multitasking operating system, stalls while waiting for I/O, and other moments when you code is not actively working.

Answer 2

There are many things going on in your system other than running your Ruby code. Elapsed time is the total real time taken and should not be used for benchmarking. You want the system and user CPU times since those are the times that your process actually had the CPU.

An example, if your process:

• used the CPU for one second running your code; then
• used the CPU for one second running OS kernel code; then
• was swapped out for seven seconds while another process ran; then
• used the CPU for one more second running your code,

you would have seen:

• ten seconds elapsed time,
• two seconds user time,
• one second system time,
• three seconds total CPU time.

The three seconds is what you need to worry about, since the ten depends entirely upon the vagaries of the process scheduling.

Answer 3

It may also be the case that the CPU time when your code is executing is not counted.

The extreme example is a real-time system where the timer triggers some activity which is always shorter than a timer tick. Then the CPU time for that activity may never be counted (depending on how the OS does the accounting).

Answer 4

You don't want to totally discount clock-on-the-wall time. Time used to wait w/o another thread ready to utilize CPU cycles may make one piece of code less desirable than another. One set of code may take some more CPU time, but, employ multi-threading to dominate over the other code in the real world. Depends on requirements and specifics. My point is ... use all metrics available to you to make your decision.

Also, as a good practice, if you want to compare two pieces of code you should be running as few extraneous processes as possible.