Performance profiling on Linux

Question

What are the best tools for profiling C/C++ applications on *nix?

(I'm hoping to profile a server that is a mix of (blocking) file IO, epoll for network and fork()/execv() for some heavy lifting; but general help and more general tools are all also appreciated.)

Can you get the big system picture of RAM, CPU, network and disk all in one overview, and drill into it?

There's been a lot of talk on the kernel lists about things like perf timechart, but I haven't found anything turning up in Ubuntu yet.

Answer 1

Compile with -pg, run the program, and then use gprof

Compiling (and linking) with -pg adds profiling code and the profiling libraries to the executable, which then produces a file called gmon.out that contains the timing information. gprof displays call graphs and their (absolute and relative) timings.

See man gprof for details.

Answer 2

Description of using -gp and gproff here http://www.ibm.com/developerworks/library/l-gnuprof.html

Answer 3

If you can take your application to freeBSD, OS X , or Solaris you can use dtrace, although dtrace is an analyst oriented tool -- i.e., you need to drive it -- read: script it. Nothing else can give you the level of granularity you need; Dtrace can not just profile the latencies of function calls in user-land; it can also follow a context switch into the kernel.

Answer 4

The FOSS answer, as already mentioned, is to build with -pg and then use gprof to analyse the output. If it's a product/project that justifies throwing some money at, I would also be tempted to use IBM/Rationals Quantify profiler as that makes it easier to view the profiling data, drill down to the line level or look at it in a '10000ft' level.

Of course there might be viewer for gprof available that can do the same thing, but I am not aware of any.

Answer 5

oprofile might interest you. Ubuntu should have all the packages you need.

Answer 6

The canonical example of a full system profiling tool (for Solaris, OS X, FreeBSD) is DTrace. But it is not yet fully available on Linux (you can try here but the site is down for me at the moment, and I haven't tried it myself). There are many tools, in various states of usefulness, for doing full system profiling and kernel profiling on Linux.

You might consider investigating:

• oprofile
• SystemTap
• bootchart
• strace (e.g. this SO answer

Answer 7

I recommend taking stackshots, for which pstack is useful. Here's some more information:

1. Comments on gprof.

2. How stackshots work.

3. A blow-by-blow example.

4. A very short explanation.

If you want to spend money, Zoom looks like a pretty good tool.

Answer 8

For performance, you can try Callgrind, a Valgrind tool. Here is a nice article showing it in action.