I need to find the point in my userland code that crash my kernel.

Question

I have big system that make my system crash hard. When I boot up, I don't even have a coredump. If I log every line that get executed until my system goes down. I will find that evil code.

Can I log every source code line in GDB to a file?

UPDATE:

ok, I found the bug. It was nasty. The application I started did not take the system down. After learning about coredump inspection with mdb, and some gdb stepping I found out that the systemcall causing the dump, was not implemented. Updating the system to latest kernel will fix my problem. Thanks to all of you.

MY LESSON:

make sure you know what process causes the coredump. It's not always the one you started.

Answer 1

Sounds like a tricky little problem.

I often try to eliminate as many possible suspects as I can by commenting out large chunks of code, configuring the system to not run certain pieces (if it allows you to do that) etc. This amounts to doing an ad-hoc binary search on the problem, and is a surprisingly effective way of zooming in on offending code relatively quickly.

A potential problem with logging is that the log might not hit the disk before the system locks up - if you don't get a core dump, you might not get the log.

Speaking of core dumps, make sure you don't have a limit on your core dump size (man ulimit.)

You could try to obtain a list of all the functions in your code using objdump, process it a little bit and create a bunch of GDB trace statements on those functions - basically creating a GDB script automatically. If that turns out to be overkill, then a binary search on the code using tracepoints can also help you zoom in on the problem.

And don't panic. You're smarter than the bug - you'll find it.

Answer 2

I'm not familiar with the gdb way of doing this, but with windbg the way to go is to have a debugger attached to the kernel and control the debugger remotely over a serial cable (or firewire) from a second debugger. I'm pretty sure gdb has similar capabilities, I could quickly find some hints here: http://www.digipedia.pl/man/gdb.4.html

Answer 3

Please clarify your problem: What part of the system is crashing?

Is it an application? If so, which application? Is this an application which you have written yourself? Is this an application you have obtained from elsewhere? Can you obtain a clean interrupt if you use a debugger? Can you obtain a backtrace showing which functions are calling the section of code which crashes?

Is it a new hardware driver? Is it based on an older driver? If so, what has changed? Is it based on a manufacturer's data sheet? Is that data sheet the latest and most correct?

Is it somewhere in the kernel? Which kernel?

What is the OS? I assume it is linux, seeing that you are using the GNU debugger. But of course, that is not necessarily so.

You say you have no coredump. Have you enabled coredumps on your machine? Most systems these days do not have coredumps enabled by default.

Regarding logging GDB output, you may have some success, but it depends where the problem is whether or not you will have the right output logged before the system crashes. There is plenty of delay in writing to disk. You may not catch it in time.

Answer 4

You can not reasonably track every line of your source using GDB (too slow). Besides, a system crash is most likely a result of a system call, and libc is probably doing the system call on your behalf. Even if you find the line of the application that caused OS crash, you still don't really know anything.

You should start by clarifying which OS is crashing. For Linux, you can try the following approaches:

strace -fo trace.out /path/to/app

After reboot, trace.out will contain syscalls the application was doing just before the crash. If you are lucky, you'll see the last syscall-of-death, but I wouldn't count on it.

Alternatively, try to reproduce the crash on the user-mode Linux, or on kernel with KGDB compiled in.

These will tell you where the problem in the kernel is. Finding the matching system call in your application will likely be trivial.