Profiling of Python threads

Question

Hello,

I am trying to figure out how to measure the performance of several python threads in my application. I currently have several tasks that are executing on different threads based on user input and I would like to measure the execution time, maybe even memory consumption of each of the threads. I have tried to use cProfile (on each instantiation of the thread, then I would write the data to a file and then aggregate all results) with limited success. Also, I have an added problem of having some blocking IO which is skewing my results. Is there anyway to effectively profile my application?

Answer 1

There are a few different ways to tackle this problem. cProfile is great and comes with Python, however many people see multi-threaded profiling as an issue. One way of getting around this is by running separate instances of cProfile for each thread and then combining the results using Stats.add.

Should that not be as useful as you'd hoped, another alternative could be to use Yappi, which I've had success using for a few special multi-threaded cases. It's got great documentation so you shouldn't have too much trouble setting it up.

For memory specific profiling, check out Heapy. But be warned, it may create some of the largest log files you've ever seen if your code is bad!