Books about/for deeper understanding debugging C or C++ (assembly) on Windows?

Question

I'm a professional C++ programmer and I'm pretty OK with the Visual Studio debugger (currently using 2005). I halfway know what a processor register is supposed to be, I can make something of a disassembly view (mostly guessing at what individual asm instructions might do). I know about symbol files -- that is their existence and how to create and use them, without any theoretical background. From time to time I'll try WinDbg to make something of a MiniDump file that seems to be broken and more often than not I get completely lost in it's horrible UI.

I have the book Advanced Windows Debugging (Hewardt, Pravat) and while it's been an interesting read so far it's a bit lacking in providing sufficient background to the techniques they explain. Paul mentions the IDA Pro Book in this answer as providing some insights, although I'm a bit sceptical if it really provides background.

I have the feeling that one key to being able to understand whats going on when debugging and jumping through C++ disassembly is to have a good understanding about how the (x86) assembly constructs relate to the C++ code, i.e. knowing what the compiler generates, what registers it uses for what, etc.

So, what books would you recommend to help a C++ programmer get a deeper knowledge of what's going on when debugging?

Answer 1

I'd love to point you to a single book or site and tell you to go forth, but unfortunately it's a bit trickier than that. I think that most people tend to learn the basic skill set over time as part of their job then want to write about techniques from that point on because that's what's interesting. So, you don't end up with much starter material. It is a skill set that can be taught and learned though, so don't feel too bad yet :) And chicks dig dudes that can debug...

The keys to becoming a good analyst are:

1) Understanding your operating environment. This includes the O/S that runs your code as well as the processor architecture on which it runs.

2) Understanding your tools. If you're going to use WinDBG, you have to go through the pain and learn it.

So, what type of debugging are you're expecting to do? While user mode debugging and kernel mode debugging are similar at the core, the environment is sufficiently different to require different base knowledge tracks. For example, I've been using WinDBG for over 10 years doing kernel debugging and I just scratch my head at all of the .NET questions asked here.

If you're doing kernel debugging, the Windows Internals book is an absolute must:

http://technet.microsoft.com/en-us/sysinternals/bb963901.aspx

Read it. Twice. Yes, it will be boring. Yes, you will get strange looks on the train and possibly strain your back lugging it around. But, you don't have a choice :)

For user mode I'm less help, maybe someone has ideas. I'd still recommend the Windows Internals book though.

For either, I highly suggest this very old but still excellent article on the x86:

http://www.microsoft.com/msj/0298/hood0298.aspx

It explains 99% of the concepts you need to know to do assembly level debugging on the x86. The x64 is enough of a deviation that you'll need to learn that separately, but once you understand the x86 you'll be good to go.

Unless you want to get really hardcore, stay away from the ASM books out there. They exist to teach you how to write assembly, which is not at all what you need. Reading assembly language is much, much easier and only requires a basic understanding and practice. The good news there as well is that most compilers spit out the same code over and over, so you don't even need to learn that many instructions.

Visual Studio has an option where it will overlay the ASM with your source code, so that you can see the assembly generated for a particular line of code. This is a good exercise if you're just starting out as you can see how code breaks down to ASM.

The Advanced Windows Debugging book is impressive, but it really is advanced. Those dudes scare ME...

And, above all, the most important thing to do is practice. Debugging is self reinforcing, the more you do the better you get at it. So, grabbing dumps and trying to solve them is the best way to learn this. If you can't figure them out and don't have the luxury of asking someone down the hall, ask for help. Once you solve one you'll be hooked and ready for the next one :)

I'll also mention that there are courses out there for debugging. I teach them for kernel debugging (with and without IDA):

http://www.osr.com/debug.html

http://www.osr.com/debug_idapro.html

Students come in on Monday without ever having heard of WinDBG and leave on Friday debugging a real life system crash, which is pretty cool. I'm sure similar classes exist for user mode debugging.

Good luck and welcome to the group!

-scott

Answer 2

When talking about advanced Windows stuff, I immediately think of Mark Russinovich (Winternals/SysInternals).

He has a great blog : Mark's blog and he has written some books for Microsoft Press : here is his latest book.

Answer 3

Not a book per-se but the bugslayer columns from MSDN contain a wealth of information, even those going back 15 years are still useful for understanding the basic concepts of what is going on under the hood.

See http://www.google.com/search?hl=en&q=Bugslayer+site:microsoft.com

John Robbin, the author of a lot (all?) of the columns also has a couple of debugging books which are worthwhile: http://www.amazon.com/John-Robbins/e/B001IXMLF0/

Answer 4

The other answers probably have some great resources. Here's my addition.

I've gone through pretty much that same as you. Sadly, I don't have a particular resource that covers everything. I've learned just enough about WinDBG to make able to make some use of dump files, but that was mostly through a bunch of searching various articles on Google.

However, one thing that helped me get a grasp on assembly language, registers, and how all that relates to C++ was this free e-book called PC Assembly Language. It's specifically about 32-bit (x86) assembly, and even has a chapter how things relate to various C++ concepts.