Post-mortem crash-dump debugging without having the exact version of a Windows DLL in the Symbol Server

Question

Within my application, I use the MiniDumpWriteDump function (see dbghelp.dll) to write a crash dump file whenever my application crashes.

I also use a symbol server to store all my executables and pdb files, so that whenever a customer sends me a crash-dump file, the debugger automatically picks up the correct version of the executable and the debug information.

I also store Windows DLL's (ntdll.dll, kernel32.dll, ...) and their debug information in the symbol server (using SymChk). The debug information is fetched from Microsoft's public symbol server.

Most of the time this works perfect, except when:

• a customer crashes in one of the Windows DLL's
• and the customer uses DLL's that I haven't put in the symbol server

This is because it is quite undoable to store every flavor of every Windows DLL in the Symbol Server (especially with the weekly patches).

So, if a customer crashes in, let's say, version 5.2.123.456 of NTDLL.DLL, and I didn't put this exact version of the DLL in my Symbol Server, then I'm stuck. Even Microsoft's public symbol server doesn't help because it only provides the debug information, not the DLL's itself.

My current solution is to ask the customer for his DLL's, but that's not always easy. Therefore I'm looking for a better solution.

Is there a way to get the debugger showing a correct call stack, or loading the debug information of a specific DLL, even if you don't have the exact version of the DLL?

Alternatively, is there a way to get all versions of all (or the important) Windows DLL's (from Microsoft)?

Answer 1

I'm pretty sure Microsoft's symbol server also provides the binaries. I'm looking in my store and I see a ton of Microsoft .dll files. I have my _NT_SYMBOL_PATH defined as

SRV*F:\Symbols\Microsoft*http://msdl.microsoft.com/download/symbols

This way it will search my local store first before trying to copy them down from Microsoft's public server.

Answer 2

? What part isn't working?

I've never been in your situation, exactly, but I would expect the debugger to give you the correct portion of the call stack that was in your code, up to the call into the dark dll. Of course, from there down to the actual crash symbols would be unavailable, but can't you see which NTDLL API was being called and which arguments were passed to that call?

You don't say which tool you're using for minidump debugging: WinDBG or VS.

Answer 3

You can have multiple symbol servers in your symbol path. So simply set up the symbol path to point to your own server for your own private modules, and to the public MS server for OS modules, see Symbol Path:

This can be easily combined with the Microsoft public symbol store by using the following initial setting:

_NT_SYMBOL_PATH=srv*c:\mysymbols*http://msdl.microsoft.com/download/symbols;cache*c:\mysymbols

The Microsoft Public Symbol Store is documented as http://msdl.microsoft.com/download/symbols.

Answer 4

It is possible to relax WinDbg's symbol resolution; see my answer to a similar question. On the other hand, the solution that I propose here relies on the fact that the DLLs are identical other than having different GUIDs identifying their debug symbols. A different version of a DLL is likely going to have a different binary, so the symbols are probably not going to match properly even if you can get them to load.