Profiling a python multiprocessing pool

Question

I'm trying to run cProfile.runctx() on each process in a multiprocessing pool, to get an idea of what the multiprocessing bottlenecks are in my source. Here is a simplified example of what I'm trying to do:

from multiprocessing import Pool
import cProfile

def square(i):
    return i*i

def square_wrapper(i):
    cProfile.runctx("result = square(i)",
        globals(), locals(), "file_"+str(i))
    # NameError happens here - 'result' is not defined.
    return result

if __name__ == "__main__":
    pool = Pool(8)
    results = pool.map_async(square_wrapper, range(15)).get(99999)
    print results

Unfortunately, trying to execute "result = square(i)" in the profiler does not affect 'result' in the scope it was called from. How can I accomplish what I am trying to do here?

Answer 1

Try this:

def square_wrapper(i):
    result = [None]
    cProfile.runctx("result[0] = square(i)", globals(), locals(), "file_%d" % i)
    return result[0]

Answer 2

If I have a multi-process system dominated by asynchronous message-passing, I use a laborious but effective method. I generate a time-stamped log of every message exchanged during a representative span of time, and figure out the full reason behind each message.

What I am looking for is:

• Messages that do not necessarily have a good reason, such as repeated requests due to timeout issues.

• Messages that show a long delay between the time they are received and the time any follow-on messages are generated. If there is such a delay, I figure out what the reason is. Was it simply queued up behind some other activity that was accessing a DB, and a simple priority-change would fix it?

Maybe there is some tool that makes this kind of thing easy. Regardless, it is heartening that asynchronous systems can be made to go quite fast.