Is creating a memory dump at customer environment good ?

Question

I am facing a severe problem with my program, which gets reproduced only in the customer place. Putting logs, are not helping as I doubt the failure is happening in a third party dll. For some reasons, I couldn't get help from the library provider. I am thinking of producing a dump at the point of failure, so that to analyze it offline. Is this a recommended practice? Or any alternatives?

Answer 1

Crash dumps are a pretty common troubleshooting method and can be very effective, especially for problems that only reproduce at the customer's site.

Just make sure the customer/client understands what you're doing and that you have permission. It's possible that a crash dump can have sensitive information that a customer may not want (or be permitted) to let walk out the door or over the wire.

Answer 2

Better than that there are libraries that will upload crash data back you.

BugDump and BugSplat

And there's the Microsoft way:

http://msdn.microsoft.com/en-us/library/aa936273.aspx

Answer 3

Disclaimer: I am not a lawyer, nor do I pretend to be one, this is not legal advice.

The data you can include in logs and crash dumps also depend on what domain you are working in. For example, medical equipment and patient information systems often contain sensitive data about patients that should not be visible to unauthorized persons.

The HIPAA Privacy Rule regulates the use and disclosure of certain information held by "covered entities" (...) It establishes regulations for the use and disclosure of Protected Health Information (PHI). PHI is any information held by a covered entity which concerns health status, provision of health care, or payment for health care that can be linked to an individual.[10] This is interpreted rather broadly and includes any part of an individual's medical record or payment history. --Wikipedia

It should not be possible to link health information to an individual. The crash dumps and logs should be anonymized and stripped of any sensitive information, or not sent at all.

Maybe this does not apply to your specific case, so this is more of a general note. I think it applies to other domains that handle sensitive information, such as military and financial, and so on.

Answer 4

Yes, this is something that every program should have and utilize as often as possible.

I suggest that you don't use third party libraries. Create your own dumps instead. It's very simple and straight forward. You basically need to do the following:

Your program needs to access dbghelp.dll. It's a windows dll that allows you to create human readable call stacks etc. The debugger uses this dll to display data in your process. It also handles post mortem debugging, i.e. dumps of some sort. This dll can safely be distributed with your software. I suggest that you download and install Debugging Tools for Windows. This will give you access to all sorts of tools and the best tool WinDbg.exe and the latest dbghelp.dll is also in that distribution.

In dbghelp.dll you call e.g. MiniDumpWriteDump(), which will create the dump file and that's more or less it. You're done. As soon as you have that file in your hands, you can start using it. Either in the Visual Studio Debugger, which probably even might be associated with the .dmp file extension, or in WinDbg.

Now, there are a few things to think of while you're at it. When checking dump files like this, you need to generate .pdb files when you compile and link your executable. Otherwise there's no chance of mapping the dump data to human readable data, e.g. to get good callstacks and values of variables etc. This also means that you have to save these .pdb files. You need to be able to match them exactly against that very release. Since the dump files are date stamped with the date stamp of the executable, the debugger needs the exact pdb files. It doesn't matter if your code hasn't changed a single bit, if the .pdb files belong to another compilation session, you're toast.

I encourage every windows win32 developer to check out Oleg Starodumov's site DebugInfo.com. It contains a lot of samples and tutorials and how you can configure and tune your dump file generation. There are of course a myriad of ways to exclude certain data, create your custom debug message to attach to the dump etc.

Keep in mind that minidumps will contain very limited information about the application state at exception time. The trade off is a small file (around 50-100 kB depending on your settings). But if you want, you can create a full dump, which will contain the state of the whole application, i.e. globals and even kernel objects. These files can be HUGE and should only be used at extreme cases.

If there are legal aspects, just make sure your customers are aware of what you're doing. I bet you already have some contract where you aren't supposed to reveal business secrets or other legal aspects. If customers complain, convince them how important it is to find bugs and that this will improve the quality of the software drastically. More or less higher quality at the cost of nothing. If it doesn't cost them anything, that's also a good argument :)

Finally, here's another great site if you want to read up more on crash dump analysis: dumpanalysis.org

Hope this helps. Please comment if you want me to explain more.

Cheers !

Edit:

Just wanted to add that MiniDumpWriteDump() requires that you have a pointer to a MINIDUMP-EXCEPTION-INFORMATION (with underscores) struct. But the GetExceptionInformation() macro provides this for you at time of exception in your exception handler (structured exception handling or SEH):

__try {

}
__except (YourHandlerFunction(GetExceptionInformation())) {

}

YourHandlerFunction() will be the one taking care of generating the minidump (or some other function down the call chain). Also, if you have custom errors in your program, e.g. something happens that should not happen but technically is not an exception, you can use RaiseException() to create your own.

GetExceptionInformation() can only be used in this context and nowhere else during program execution.

Answer 5

Basically the easiest way to produce a dumpfile is by using adplus. You don't need to change your code.

Adplus is part of the debugging tools for windows, as mentioned in the article above. Adplus is basically a huge vbscript automation of windbg.

What you have to do to use adplus:

1. Download and install Debugging tools for windows to c:\debuggers
2. start your application
3. open a commandline and navigate to c:\debuggers
4. run this line "adplus -crash your_exe.exe"
5. reproduce the crash

you'll get a minidump with all the information you need. you can open the crash dump in your favorite debugger. within windbg, the command "analyze -v" helped me in at least 40% of all the crashes that only happened at customer site and were not reproducible in house.