How to use gprof to profile a daemon process without terminating it gracefully?

Question

Need to profile a daemon written in C++, gprof says it need to terminate the process to get the gmon.out. I'm wondering anyone has ideas to get the gmon.out with ctrl-c? I want to find out the hot spot for cpu cycle

Answer 1

As a first suggestion I would say you might try to use another tool. If the performance of that daemon is not an issue in your test you could give a try to valgrind. It is a wonderful tool, I really love it.

Answer 2

Need to profile a daemon written in C++, gprof says it need to terminate the process to get the gmon.out.

That fits the normal practice of debugging daemon processes: provision a switch (e.g. with command line option) which would force the daemon to run in foreground.

I'm wondering anyone has ideas to get the gmon.out with ctrl-c?

I'm not aware of such options.

Though in case of gmon, call to exit() should suffice: if you for example intend to test say processing 100K messages, you can add in code a counter incremented on every processed message. When the counter exceeds the limit, simply call exit().

You also can try to add a handler for some unused signal (like SIGUSR1 or SIGUSR2) and call exit() from there. Thought I do not have personal experience and cannot be sure that gmon would work properly in the case.

I want to find out the hot spot for cpu cycle

My usual practice is to create a test application, using same source code as the daemon but different main() where I simulate precise scenario (often with a command line switch many scenarios) I need to debug or test. For the purpose, I normally create a static library containing the whole module - except the file with main() - and link the test application with the static library. (That helps keeping Makefiles tidy.)

I prefer the separate test application to hacks inside of the code since especially in case of performance testing I can sometimes bypass or reduce calls to expensive I/O (or DB accesses) which often skews the profiler's sampling and renders the output useless.

Answer 3

If you want to make the daemon go as fast as possible, you can use lsstack with this technique. It will show you what's taking time that you can remove. If you're looking for hot spots, you are probably looking for the wrong thing. Typically there are function calls that are not absolutely needed, and those don't show up as hot spots, but they do show up on stackshots.

Another good option is RotateRight/Zoom.