Accurate timing of functions in python

Question

I'm programming in python on windows and would like to accurately measure the time it takes for a function to run. I have written a function "time_it" that takes another function, runs it, and returns the time it took to run.

   def time_it(f, *args):
       start = time.clock()
       f(*args)
       return (time.clock() - start)*1000

i call this 1000 times and average the result. (the 1000 constant at the end is to give the answer in milliseconds.)

This function seems to work but i have this nagging feeling that im doing something wrong, and that by doing it this way im using more time than the function actually uses when its running.

Is there a more standard or accepted way to do this?

When i changed my test function to call a print so that it takes longer, my time_it function returns an average of 2.5 ms while the cProfile.run('f()') returns and average of 7.0 ms. I figured my function would overestimate the time if anything, what is going on here?

One additional note, it is the relative time of functions compared to each other that i care about, not the absolute time as this will obviously vary depending on hardware and other factors.

Answer 1

Instead of writing your own profiling code, I suggest you check out the built-in Python profilers (profile or cProfile, depending on your needs): http://docs.python.org/library/profile.html

Answer 2

Use the timeit module from the Python standard library.

Basic usage:

from timeit import Timer

# first argument is the code to be run, the second "setup" argument is only run once,
# and it not included in the execution time.
t = Timer("""x.index(123)""", setup="""x = range(1000)""")

print t.timeit() # prints float, for example 5.8254
# ..or..
print t.timeit(1000) # repeat 1000 times instead of the default 1million

Answer 3

This code is very inaccurate

total= 0
for i in range(1000):
    start= time.clock()
    function()
    end= time.clock()
    total += end-start
time= total/1000

This code is less inaccurate

start= time.clock()
for i in range(1000):
    function()
end= time.clock()
time= (end-start)/1000

The very inaccurate suffers from measurement bias if the run-time of the function is close to the accuracy of the clock. Most of the measured times are merely random numbers between 0 and a few ticks of the clock.

Depending on your system workload, the "time" you observe from a single function may be entirely an artifact of OS scheduling and other uncontrollable overheads.

The second version (less inaccurate) has less measurement bias. If your function is really fast, you may need to run it 10,000 times to damp out OS scheduling and other overheads.

Both are, of course, terribly misleading. The run time for your program -- as a whole -- is not the sum of the function run-times. You can only use the numbers for relative comparisons. They are not absolute measurements that convey much meaning.