System-wide continuous linux library profiling

Question

Let's say I want to profile usage of a single library which is used in many apps in the system. I'm ok with only statistical profile, not an accurate one (but that one would be nice too).

What I do not want to do is to recompile every running program with profiling support. I'd like the profiling information to be dumped either periodically or on demand to some file. It would be really good if the stats were broken down using application names, or pids.

Is there any way to achieve that right now?

Example usage: profiling the glib library using the data from a running system.

Answer 1

Look at oprofile. I think it does exactly what you want.

Oprofile uses hardware counter sampling to profile the code, and you can customize which counter you want to sample on (e.g. if you don't want a time profile, you could sample on the floating point instruction counter and see the parts of your code that do the most FP work). For time profiles, the rate is ~2000 samples per second, so the overhead is very low.

Summary data generated tells you the application, load module, and symbols (if you had symbol info) where you spent most of your time. You can also opt to profile kernel- and user-space separately, and callpath information is available if you want. The latest version of oprofile even has support for profiling JITed code, so it's pretty comprehensive.