I want to know how does gdb work internally. e.g. I know a brief idea that it makes use of ptrace() system call to monitor traced program. But I want to know how it handles signals, how it inserts new code, and other such fabulous things it does.
The only way you'll find out is by studying the source.
You can also build it and debug it with itself. Step through the code, and you'll know exactly how it does what it does.
Reading GDB source is not for the faint of heart though -- it is chock-full of macros, and heavily uses libbfd
, which itself is hard to understand.
It has to, because it is portable (and in particular, builds and works on platforms which do not have ptrace()
at all).
Check out this pdf it covers some of the important aspects of GDB internals.
From the pdf:
This document documents the internals of the GNU debugger, gdb. It includes descrip- tion of gdb’s key algorithms and operations, as well as the mechanisms that adapt gdb to specific hosts and targets.
You may want to take a look at the libthread_db
interface, which is the main interface (with proc_service
) between thread libraries and debuggers.
This article and its presentation detail how the processes interact with each other, which is useful to understand which part of the thread handling GDB is reliable for, and which part is 'out-sourced' to the thread library: