Profiling Python generators

Question

I'm adapting an application that makes heavy use of generators to produce its results to provide a web.py web interface.

So far, I could wrap the call to the for-loop and the output-producing statements in a function and call that using cProfile.run() or runctx(). Conceptually:

def output():
    for value in generator():
        print(value)

cProfile.run('output()')

In web.py, I have to wrap it the following way, since I want to immediately produce output from the potentially long-running computation in each iteration step using yield:

class index:
    def GET(self):
        for value in generator():
            yield make_pretty_html(value)

Is there a way to profile all calls to the generator like in the first example when it's used like in the second one?

Answer 1

Can you just use time.time() to profile the parts you are interested in? Just get the current time and subtract from the last time you made a measurement.

Answer 2

It sounds like you're trying to profile each call to 'next' on the generator? If so, you could wrap your generator in a profiling generator. Something like this, where the commented off part will be sending the results to a log or database.

def iter_profiler(itr):
  itr = iter(itr)
  while True:
    try:
      start = time.time()
      value = itr.next()
      end = time.time()
    except StopIteration:
      break
    # do something with (end - stop) times here
    yield value

Then instead of instantiating your generator as generator() you would use iter_profiler(generator())

Answer 3

I finally found a solution. Return value of profiling via here.

import cProfile
import pstats
import glob
import math

def gen():
    for i in range(1, 10):
        yield math.factorial(i)

class index(object):
    def GET(self):
        p = cProfile.Profile()

        it = gen()
        while True:
            try:
                nxt = p.runcall(next, it)
            except StopIteration:
                break
            print nxt

        p.print_stats()

index().GET()

I also could merge multiple such profiling results (once I start giving unique file names) via documentation and store/analyze them combined.