How to profile multi-threaded C++ application on Linux?

Question

I used to do all my Linux profiling with gprof.

However, with my multi-threaded application, it's output appears to be inconsistent.

Now, I dug this up:

http://sam.zoy.org/writings/programming/gprof.html

However, it's from a long time ago and in my gprof output, it appears my gprof is listing functions used by non-main threads.

So, my questions are:

1) In 2010, can I easily use gprof to profile multi-threaded Linux C++ applications? (Ubuntu 9.10) 2) What other tools should I look into for profiling?

Answer 1

Have a look at Valgrind.

Answer 2

Have a look at Zoom.

Answer 3

Have a look at oprofile. The profiling overhead of this tool is negligible and it works with multithreaded applications just fine.

Answer 4

A Paul R said, have a look at Zoom. You can also use lsstack, which is a low-tech approach but surprisingly effective, compared to gprof.

Added: Since you clarified that you are running OpenGL at 33ms, my prior recommendation stands. In addition, what I personally have done in situations like that is both effective and non-intuitive. Just get it running with a typical or problematic workload, and just stop it, manually, in its tracks, and see what it's doing and why. Do this several times. Now, if it only occasionally misbehaves, you would like to stop it only while it's misbehaving. That's not easy, but I've used an alarm-clock interrupt set for just the right delay. For example, if one frame out of 100 takes more than 33ms, at the start of a frame, set the timer for 35ms, and at the end of a frame, turn it off. That way, it will interrupt only when the code is taking too long, and it will show you why. Of course, one sample might miss the guilty code, but 20 samples won't miss it.