Program only crashes as release build -- how to debug?

Question

I've got a "Schroedinger's Cat" type of problem here -- my program (actually the test suite for my program, but a program nonetheless) is crashing, but only when built in release mode, and only when launched from the command line. Through caveman debugging (ie, nasty printf() messages all over the place), I have determined the test method where the code is crashing, though unfortunately the actual crash seems to happen in some destructor, since the last trace messages I see are in other destructors which execute cleanly.

When I attempt to run this program inside of Visual Studio, it doesn't crash. Same goes when launching from WinDbg.exe. The crash only occurs when launching from the command line. This is happening under Windows Vista, btw, and unfortunately I don't have access to an XP machine right now to test on.

It would be really nice if I could get Windows to print out a stack trace, or something other than simply terminating the program as if it had exited cleanly. Does anyone have any advice as to how I could get some more meaningful information here and hopefully fix this bug?

Edit: The problem was indeed caused by an out-of-bounds array, which I describe more in this post. Thanks everybody for your help in finding this problem!

Answer 1

When I have encountered problems like this before it has generally been due to variable initialization. In debug mode, variables and pointers get initialized to zero automatically but in release mode they do not. Therefore, if you have code like this

int* p;
....
if (p == 0) { // do stuff }

In debug mode the code in the if is not executed but in release mode p contains an undefined value, which is unlikely to be 0, so the code is executed often causing a crash.

I would check your code for uninitialized variables. This can also apply to the contents of arrays.

Answer 2

Something similar happend to me once with GCC. It turned out to be a too aggressive optimization that was enabled only when creating the final release and not during the development process.

Well, to tell the truth it was my fault, not gcc's, as I didn't noticed that my code was relying on the fact that that particular optimization wouldn't have been done.

It took me a lot of time to trace it and I only came to it because I asked on a newsgroup and somebody made me think about it. So, let me return the favour just in case this is happening to you as well.

Answer 3

Crashes like this are almost always caused because an IDE will usually set the contents of uninitialized variable to zeros, null or some other such 'sensible' value, whereas when running natively you'll get whatever random rubbish that the system picks up.

Your error is therefore almost certainly that you are using something like you are using a pointer before it has been properly initialized and you're getting away with it in the IDE because it doesn't point anywhere dangerous - or the value is handled by your error checking - but in release mode it does something nasty.

Answer 4

You can set WinDbg as your postmortem debugger. This will launch the debugger and attach it to the process when the crash occurs. To install WinDbg for postmortem debugging, use the /I option (note it is capitalized):

windbg /I

More details here.

As to the cause, it's most probably an unitialized variable as the other answers suggest.

Answer 5

I've found this this article useful for your scenario. ISTR the compiler options were a little out of date. Look around your Visual Studio project options to see how to generate pdb files for your release build, etc.

Answer 6

With regard to your problems getting diagnostic information, have you tried using adplus.vbs as an alternative to WinDbg.exe? To attach to a running process, use

adplus.vbs -crash -p <process_id>

Or to start the application in the event that the crash happens quickly:

adplus.vbs -crash -sc your_app.exe

Full info on adplus.vbs can be found at: http://support.microsoft.com/kb/286350

Answer 7

In order to have a crash dump that you can analyze:

1. Generate pdb files for your code.
2. You rebase to have your exe and dlls loaded in the same address.
3. Enable post mortem debugger such as Dr. Watson
4. Check the crash failures address using a tool such as crash finder.

You should also check out the tools in Debugging tools for windows. You can monitor the application and see all the first chance exceptions that were prior to your second chance exception.

Hope it helps...

Answer 8

In 100% of the cases I've seen or heard of, where a C or C++ program runs fine in the debugger but fails when run outside, the cause has been writing past the end of a function local array. (The debugger puts more on the stack, so you're less likely to overwrite something important.)

Answer 9

It's suspicious that it would happen outside the debugger and not inside; running in the debugger does not normally change the application behavior. I would check the environment differences between the console and the IDE. Also, obviously, compile release without optimizations and with debug information, and see if that affects the behavior. Finally, check out the post-mortem debugging tools other people have suggested here, usually you can get some clue from them.

Answer 10

Even though you have built your exe as a release one, you can still generate PDB (Program database) files that will allow you to stack trace, and do a limited amount of variable inspection. In your build settings there is an option to create the PDB files. Turn this on and relink. Then try running from the IDE first to see if you get the crash. If so, then great - you're all set to look at things. If not, then when running from the command line you can do one of two things:

1. Run EXE, and before the crash do an Attach To Process (Tools menu on Visual Studio).
2. After the crash, select the option to launch debugger.

When asked to point to PDB files, browse to find them. If the PDB's were put in the same output folder as your EXE or DLL's they will probably be picked up automatically.

The PDB's provide a link to the source with enough symbol information to make it possible to see stack traces, variables etc. You can inspect the values as normal, but do be aware that you can get false readings as the optimisation pass may mean things only appear in registers, or things happen in a different order than you expect.

NB: I'm assuming a Windows/Visual Studio environment here.

Answer 11

I agree with Rolf. Because reproducibility is so important, you shouldn't have a non-debug mode. All your builds should be debuggable. Having two targets to debug more than doubles your debugging load. Just ship the "debug mode" version, unless it is unusable. In which case, make it usable.

Answer 12

Things to look out for:

Array overruns - the visual studio debugger inserts padding which may stop crashes.

Race conditions - do you have multiple threads involved if so a race condition many only show up when an application is executed directly.

Linking - is your release build pulling in the correct libraries.

Things to try:

Minidump - really easy to use (just look it up in msdn) will give you a full crash dump for each thread. You just load the output into visual studio and it is as if you were debugging at the time of the crash.

Answer 13

Debugging release builds can be a pain due to optimizations changing the order in which lines of your code appear to be executed. It can really get confusing!

One technique to at least narrow down the problem is to use MessageBox() to display quick statements stating what part of the program your code has got to ("Starting Foo()", "Starting Foo2()"); start putting them at the top of functions in the area of your code that you suspect (what were you doing at the time when it crashed?). When you can tell which function, change the message boxes to blocks of code or even individual lines within that function until you narrow it down to a few lines. Then you can start printing out the value of variables to see what state they are in at the point of crashing.

Answer 14

Ntdll.dll with debugger attached

One little know difference between launching a program from the IDE or WinDbg as opposed to launching it from command line / desktop is that when launching with a debugger attached (i.e. IDE or WinDbg) ntdll.dll uses a different heap implementation which performs some little validation on the memory allocation/freeing.

You may read some relevant information in unexpected user breakpoint in ntdll.dll. One tool which might be able to help you identifying the problem is PageHeap.exe.

Crash analysis

You did not write what is the "crash" you are experiencing. Once the program crashes and offers you to send the error information to the Microsoft, you should be able to click on the technical information and to check at least the exception code, and with some effort you can even perform post-mortem analysis (see Heisenbug: WinApi program crashes on some computers) for instructions)

Answer 15

Try using _CrtCheckMemory() to see what state the allocated memory is in . If everything goes well , _CrtCheckMemory returns TRUE , else FALSE .

Answer 16

You might run your software with Global Flags enabled (Look in Debugging Tools for Windows). It will very often help to nail the problem.

Answer 17

After many hours of debugging, I finally found the cause of the problem, which was indeed caused by a buffer overflow, caused a single byte difference:

char *end = static_cast<char*>(attr->data) + attr->dataSize;

This is a fencepost error (off-by-one error) and was fixed by:

char *end = static_cast<char*>(attr->data) + attr->dataSize - 1;

The weird thing was, I put several calls to _CrtCheckMemory() around various parts of my code, and they always returned 1. I was able to find the source of the problem by placing "return false;" calls in the test case, and then eventually determining through trial-and-error where the fault was.

Thanks everybody for your comments -- I learned a lot about windbg.exe today! :)

Answer 18

Make your program generate a mini dump when the exception occurs, then open it up in a debugger (for example, in WinDbg). The key functions to look at: MiniDumpWriteDump, SetUnhandledExceptionFilter

Answer 19

Vista SP1 actually has a really nice crash dump generator built into the system. Unfortunately, it isn't turned on by default!

See this article: http://msdn.microsoft.com/en-us/library/bb787181%28VS.85%29.aspx

The benefit of this approach is that no extra software needs to be installed on the affected system. Grip it and rip it, baby!