The Community-Driven GDB Primer

Question

I was reading this question and realized it might be helpful for entry- and pro-level developers alike (including myself) to have a common reference for best practices in using gdb.

Many questions asked on Stack Overflow could easily be solved by taking some time to step some code in a debugger, and it would be good to have a community-approved resource to "teach them how to fish", so to speak. Even for those seasoned veterans who occasionally find themselves in gdb when they are accustomed to a GUI-tastic debugger might benefit from those who are much more familiar with the command line tool.

For starters (both to gdb and to prime this thread) I submit:

• Ninefinger's gdb primer
• The gdb quick reference guide, which is useful for telling you what commands are available but not how best to use them.

My hope is this thread is a seed planted that is of continued value to the community. If by "continued value" the community decides to nix it altogether, well then the masses have spoken.

Answer 1

Simple trick to find the source of a memory corruption using gdb.

In languages such as C if you are not careful you can endup with some headache causing memory corruptions (eg: writing too much into a string which end's up writing in the next value in memory)

You could use tools like valgrind to detect these problems, or you can do a quick and dirty trick in gdb to find the source of the problem. All you need to do as use the watch command

Example: You have a pointer to some memory int * mem which is being randomly filled with garbage and you don't know why, then do the following.

All you need to do to find the source of this corruption is do the following

gdb> watch *mem // set a hardware watch point on the memory value that is getting corrupted
gdb> continue

Then just wait until gdb detects a change in that value. It will pin point the exact command where the memory corruption occured

Answer 2

Breakpoints: 1

There are many ways to set breakpoints in gdb. Typing help breakpoints will show you a lot of them. The most basic is the temporary breakpoint, which you can set like this:

gdb> tbreak [function name, line number or address]

You can then use

gdb> run

and gdb will break at that point, where you can step, continue or list from my "primer"

You can also use rbreak and so catch any function matching a regular expression - useful for filtering by namespaces, for example (i.e. only break in my functions).

Note: a temporary breakpoint is exactly that - temporary. Your program will break the next time you type "run", however, the time after, it will not remember your breakpoint. If you want to always break on a specific function, use a setup without the t prefix, such as catch fork() to catch all fork() commands.

Answer 3

How to set breakpoints on line number/functions/methods that get hit too many times. Sometimes you know you want to debug a function but it is called for too many cases you are not interested in.

One strategy to make gdb more selective when breaking is to use condition break points.

Say you have a function

perform_action(action *action_to_do) {
}

and for some reason it is being passed action_to_do == NULL which is causing a core dump. How do you use gdb to find the code causing this problem?

gdb> break perform_action if action_to_do == NULL
gdb> continue

gdb will not break at any occurence of perform_action except for the time action_to_do == NULL. Ofcourse you can make your conditions as complex as you want

gdb> break perform_action if (validate_action_pointer(action_to_do) == INVALID_ACTION)

where validate_action_pointer is some function defined in your code

What if you've already added your breakpoint with no conditions, and you now decided you wanted to narrow it down?

Easy

gdb> condition 1 (action_to_do == NULL) /* 1 is your breakpoint number */

To know which breakpoint is which breakpoint number you can do

gdb> info break 

This will list all your defined breakpoints and the number for them

Answer 4

How to make your program ignore SIGNALS

If your debugging a GUI application on linux (at least X11 applications) you might get frustrated by all the SIGUSR1, SIGUSR2 which are interrupting your debugging session.

here is a simple way in gdb to stop them interrupting your session.

gdb> handle SIGUSR1 nostop noprint

You can chose what ever signal instead of SIGUSR1 and you can chose to have it printing (use print instead of noprint) and the same for stopping

Answer 5

Backtrace

A really useful command in the context of my previous answer (and recursion); whenever you've broken, type:

gdb> backtrace
#0  rec (n=8) at x.c:10
#1  0x00000000004004fc in rec (n=9) at x.c:11
#2  0x00000000004004fc in rec (n=10) at x.c:11
#3  0x00000000004004d7 in main () at x.c:5

And see output as shown - the stack trace of functions being called, files and line numbers if you're not sure what's going on - useful for exploring large codebases.

For lazy programmers out there you can also just type bt.

You can also limit the number of stack frames to show, e.g.:

gdb> backtrace 3

will only show the top three frames of the stack.

Answer 6

How to read the contents of a core dump in gdb

Too easy

gdb> file your_executable_file_path
gdb> core your_core_dump
gdb> where /* will give you the stack trace where the core dump happened and the SIGNAL that triggered the coredump SIGABORT for example */
gdb> display blah /* now you can inspect variable blah to see the state of the memory at the time of the crash */

Answer 7

display

One of the most underappreciated basic features is display. Every time the execution stops you can get a quick preview of what's going on with counters, variables, pointer values, contents of arrays, arguments of a function and anything else in memory.

(gdb) break myfunction
(gdb) c
(gdb) display k
1: k = 1
(gdb) display arg1
2: arg1 = 8
(gdb) c
Breakpoint 1, myfunction (arg1=9) at myfile.c:130
130      /* first executable line of code in myfunc's body */
2: arg1 = 9
1: k = 4

You can set it up and just hit return until you find an interesting situation. I'm always surprised how many heavy gdb users don't take advantage of this feature and run a print manually on every stop. Note the expression doesn't need to be just a variable, play with it to show different things on different breakpoints.

Answer 8

How to debug multi-threaded programs

You can display thread information with info thread

(gdb) info thread
 4 Thread 0x4251d950 (LWP 29491)  0x55f7fe1d in pthread_cond_timedwait()
   from /lib/libpthread.so.0
  3 Thread 0x41d1c950 (LWP 29490)  0x55f7fe1d in pthread_cond_timedwait()
   from /lib/libpthread.so.0
  2 Thread 0x40f97950 (LWP 29489)  0x55ceb118 in epoll_wait ()
   from /lib/libc.so.6
  1 Thread 0x7fac572cf700 (LWP 29486)  0x55f7fb99 in pthread_cond_wait()
   from /lib/libpthread.so.0

This tells you where each of your thread is executing and can help you debug deadlocks, for example.

To switch to a particular thread use the thread command.

(gdb) thread 2
[Switching to thread 2 (Thread 0x40f97950 (LWP 29489))]#0
 0x00007fac55ceb118 in epoll_wait ()
   from /lib/libc.so.6
(gdb) bt
#0  0x55ceb118 in epoll_wait () from /lib/libc.so.6
#1  0x56a817ec in ServerThread (arg=0x6e2880)
    at /home/src/srv/srv.c:1588

Now you can debug this thread by itself.

You can also execute a command on multiple threads simultaneously with the thread apply command. For example, thread apply all backtrace will apply the backtrace command to all the currently running threads. thread apply 1 3 info args will apply the info args command to threads 1 and 3, and so on.

You may run into problems if you try to step through your execution with the step or next command, because other threads will resume running, possibly throwing you out of the thread you are interested in examining. To avoid that, use the set scheduler-locking command. When set to on, other threads will not be able to run while you are examining your current thread.

Answer 9

Using GDB with Assembly

This from pm100's answer for a question about the "casual assembly reader":

I would start by following in gdb at run time; you get a better feel for whats happening. But then maybe thats just me. it will disassemble a function for you (disass func) then you can single step through it

The important bit in here is that when you are stepping through code / have hit a breakpoint, you can do:

gdb> disass func

And see how it works in assembler. I wish I knew this sooner!

Answer 10

Setting command to debug incl. parameters from the command line

In case you have a program you call like this:

$ ./prog -a -b foobar

you can invoke gdb directly using

$ gdb --args ./prog -a -b foobar

and gdb will populate the arguments, so you might safe some copy&pasting or such and directly run the command.