How to run gdb against a daemon in the background?

Question

I'm trying to debug a server I wrote with gdb as it segfaults under very specific and rare conditions.

Is there any way I can make gdb run in the background (via quiet or batch mode?), follow children (as my server is a daemon and detaches from the main PID) and automatically dump the core and the backtrace (to a designated file) once the program crashes?

Answer 1

I'm not really a gdb expert but two things come to mind

1. Tracepoints which might give you the necessary information as your program runs or
2. Use gdb's remote debugging facility to debug your program while it's running as a daemon.

Answer 2

http://stackoverflow.com/questions/77005/how-to-generate-a-stacktrace-when-my-gcc-c-app-crashes answer for this question should do what you want. (assuming you can make changes in your code)

Answer 3

First, I'd setup your shell / environment to give you a core dump. In bash:

ulimit -c unlimited

Once you have the core dump, you can use gdb to examine the stack trace:

gdb /path/to/app /path/to/core/file

Answer 4

Why not just run the process interactively in a persistent screen session? Why must it be a daemon when debugging? Or just run gdb in the screen session and attach it to the running process (e.g. gdb /path/to/binary -p PID_of_binary) after it forks.

Answer 5

Assuming you have appropriate permissions, you can have gdb attach to any process. You can do it on the command line with:

gdb /path/to/binary _pid_

or from within gdb with the attach command:

attach _pid_

So, once your daemon has started, you can use either of these techniques to attach to the final PID your daemon is running as. Attaching gdb stops the process which you are tracing so you will need to issue a "continue" to restart it.

I don't know a direct way to get gdb to run arbitrary commands when the program crashes. Here is one workaround I can think of:

1. Create and register a signal handlers for SIGSEGV.
2. Tell gdb not to stop on that signal (handle SIGSEGV nostop)
3. Set a breakpoint at the first line of your signal handler.
4. Assign commands to the breakpoint from step 3

Answer 6

You might want to take a look at how Samba facilitates debugging; it has a configurable "panic action" that can suspend the application, notify the developer, spawn gdb, etc., and is run as part of its signal handler. See lib/util/fault.c in the Samba source tree.