I'm developing an inherently multithreaded module in Python, and I'd like to find out where it's spending its time. cProfile only seems to profile the main thread. Is there any way of profiling all threads involved in the calculation?
A:
I don't know any profiling-application that supports such thing for python - but You could write a Trace-class that writes log-files where you put in the information of when an operation is started and when it ended and how much time it consumed.
It's a simple and quick solution for your problem.
Gambrinus
2009-03-17 09:16:20
+5
A:
Instead of running one cProfile, you could run separate cProfile instance in each thread, then combine the stats. Stats.add() does this automatically.
vartec
2009-03-17 09:21:43
not great when the program is starting and stopping many threads over the course of the calculation - it requires instrumenting the whole program, potentially severely affecting the results.
rog
2009-03-17 10:05:06
Actually you'd only have to modify the Thread.run() method.
vartec
2009-03-17 10:50:56
what i mean is that the overhead of creating and saving the profile instance for each thread run might easily skew the results. i don't think it's possible to make Stats without saving to a file.
rog
2009-03-17 11:48:41
Profiling shows how much active CPU time process spends in each function. It's not affected by profiler. Of course it will affect overall performance.
vartec
2009-03-17 12:09:52
1) profiling just shows time, not active cpu time (try cProfile.run('time.sleep(3)').2) Stats.add() isn't very convenient for thousands of calls (easy to run out of fds, 1000s of lines printed at start)3) overhead on thread create is factor of ~1000
rog
2009-03-17 15:52:59
+3
A:
If you're okay with doing a bit of extra work, you can write your own profiling class that implements profile(self, frame, event, arg). That gets called whenever a function is called, and you can fairly easily set up a structure to gather statistics from that.
You can then use threading.setprofile to register that function on every thread. When the function is called you can use threading.currentThread() to see which it's running on. More information (and ready-to-run recipe) here:
http://code.activestate.com/recipes/465831/
http://docs.python.org/library/threading.html#threading.setprofile
DNS
2009-03-17 13:22:20