Tools for debugging when debugger can't get you there?

Question

I have a fairly complex (approx 200,000 lines of C++ code) application that has decided to crash, although it crashes a little differently on a couple of different systems. The trick is that it doesn't crash or trap out in debugger. It only crashes when the application .EXE is run independently (either the debug EXE or the release EXE - both behave the same way). When it crashes in the debug EXE, and I get it to start debugging, the call stack is buried down into the windows/MFC part of things, and isn't reflecting any of my code. Perhaps I'm seeing a stack corruption of some sort, but I'm just not sure at the moment. My question is more general - it's about tools and techniques.

I'm an old programmer (C and assembly language days), and a relative newcomer (couple/few years) to C++ and Visual Studio (2003 for this projecT).

Are there tricks or techniques anyone's had success with in tracking down crashing issues when you cannot make the software crash in a debugger session? Stuff like permission issues, for example?

The only thing I've thought of is to start plugging in debug/status messages to a logfile, but that's a long, hard way to go. Been there, done that. Any better suggestions? Am I missing some tools that would help? Is VS 2008 better for this kind of thing?

Thanks for any guidance. Some very smart people here (you know who you are!).

cheers.

Answer 1

Might be that you have a too big object on the stack...

Explainations (from comments):

I gives this answer because that's the only case I've seen that a debuger (VS or CodeWarrior) couldn't catch and seeemed mysterious. Most of the time, that was the big application object that was defined on the stack in the main() function, and having members not allocated on the heap. Just calling new to instantiate the object fixed the obscure problem. Didn't need to get a specific tool for that in the end.

Answer 2

lint.

http://stackoverflow.com/questions/632057/c-c-free-alternative-to-lint

Answer 3

I've not done C++ professionally for over 10 years, but back in the day I used Rational PurifyPlus, which will be a good start, as is BoundsChecker (if it still exists!) These products find out of bounds accesses, corrupted memory, corrupted stack and other problems that can go undetected until "boom" and then you have no idea where you are.

I would try these first. If that fails, then you can start typing in logging statements.

If the debugger mitigates the crash, this can be for these reasons:

• memory corruption: under a debug build memory is allocated with space before an after, so rogue writes may not corrupt under a debug session
• timing and multi-threading: the debugger alters timing of threads and can make tricky multi-threaded problems hard to nail down.

Answer 4

If it's memory corruption, a memory tracking/diagnostic tool (I used to use BoundsChecker to great effect in the good old days of C++) may help you to locate and fix the cause in minutes, where any other technique coud take days or even months.

For other cases, you've suggested another approach yourself: a sometimes labour-intensive but very effective approach to getting a "real" stack trace is to simply use printf - a vastly underrated debugging tool available in every environment. If you have a rough idea you can straddle the crash area with only a few log messages to narrow down the location, and then add more as you home in on the problem area. This can often unearth enough clues that you can isolate the cause of the crash in a few minutes, even though it can seem like a lot of work and perhaps a hopeless cause before you start.

edit:

Also, if you have the application under source control, then get a historical version from when you think it was working, and then do a binary chop between that date and "now" to isolate when the issue began to occur. This can often narrow down a bug to the precise checkin that introduced the bug, and if you're lucky it will point you at a few lines of code. (If you're unlucky the bug won't be so easily repeatable, or you'll narrow it down to a 500-file checkin where a major refactoring or similar took place)

Answer 5

I couldn't recommend more the blog of Mark Rusinovich. Absolutely brilliant guy from whom you can learn a whole bunch of debugging techniques for windows and many more. Especially try read some of the "The Case of" series! Amazing stuff!

For example take a look at this case he had investigated - a crash of IE. He shows how to capture the stack of the failing thread and many more interesting stuff. His main tools are windows debugging tools and also his sysinternals tools!

Enough said. Go read it!

Also I would recommend the book: Windows Internals 5. Again by Mark and company.

Answer 6

Get the debugging tool kit from MS ( http://www.microsoft.com/whdc/devtools/debugging/default.mspx ).

Set adplus up for crash mode monitoring ( http://www.microsoft.com/whdc/devtools/debugging/default.mspx ).

This should get you a crash dump when the app crashes. Load the dump up in WindDbg from the debugging toolkit and analyze using that. It is a painful, but very powerful, process to anaylyze out-of-debugger crashes.

There are quite a few resources around for using WinDbg - a good book on general Windows unmanaged debugging and the tools in the debugging kits is: http://www.amazon.com/Advanced-Windows-Debugging-ebook/dp/B000XPNUMW