Managing Setup code with TimeIt

Question

As part of a pet project of mine, I need to test the performance of various different implementations of my code in Python. I anticipate this to be something I do alot of, and I want to try to make the code I write to serve this aim as easy to update and modify as possible.

It's still in its infancy at the moment, but I've taken to using strings to manage common setup or testing code, eg:

naiveSetup = 'from PerformanceTests.Vectors import NaiveVector\n' \
+ 'left = NaiveVector([1,0,0])\n' \
+ 'right = NaiveVector([0,1,0])'

This allows me to only write the code once, at the expense of making it harder to read and clunky to update.

Is there a better way?

Answer 1

Use triple quotes """

setup_code = """
  from PerformanceTests.Vectors import NaiveVector
  left = NaiveVector([1,0,0])
  right = NaiveVector([0,1,0])
"""

Another interesting method is provided in the docs of timeit:

def test():
    "Stupid test function"
    L = []
    for i in range(100):
        L.append(i)

if __name__=='__main__':
    from timeit import Timer
    t = Timer("test()", "from __main__ import test")
    print t.timeit()

Though this isn't suitable for all needs.

Answer 2

Timing code is fine, but it will still leave you guessing what's going on.

To find out what's actually going on, manually pause it a few random times in the debugger, and examine the call stack.

For example, in the code that is 30x slower in one implementation than in another, each sample of the stack has a 96.7% chance of falling in the extra time that it is spending, so you can see why.

No guesswork required.