how do I stop a C++ application during execution to debug into a dll?

Question

Hi all,

I have an application for which I do not have the code, and a dll for which I do have the code. I need to be able to debug into the dll, but lacking the source for the exe, how do I do this?

The dll code is mfc c++; I believe that the main app is the same thing as well.

I've tried doing a 'set target application' deal, where I set the application that the dll will be called from, and the application crashes a horrible, horrible death when called that way. I don't know if the fault lies with this dll or with the executable for that behavior, and it's just one of the myriad things I'd like to solve.

I'm thinking that there should be some call to allow the dll to spin indefinitely until a debugger is attached to the process, at which point I should be able to debug the dll by attaching to the process. Does that make sense? Is there a better way to do this?

Answer 1

There is a registry setting called ImageFileExecutionOptions that can be set up to launch a debugger whenever your DLL is loaded. I used to use it to debug ISAPI extensions. Here is a link to a decent blog entry about it.

Answer 2

If the application is linked against a non-debug DLL and does not have debug symbols itself, this isn't really likely to be fruitful. You might want to look here for information on using windows symbol packages to help you if you're curious about what's going in inside windows DLL's, but by and large, an application which does not have debug info and which you can't compile isn't debuggable in any meaningful way.

Answer 3

With a DLL project, you should be able to tell Visual Studio to start debugging and it will ask you for an executable name. Enter your exe there. I've done this a lot for when I've worked on DLL code that was called from another process. Works for straight DLLs as well as COM components.

It might also help to set some breakpoints in your code ahead of time if you have an idea of where the problem might be.

Update: Since that does not work for you, the only other thing I can think of would be to attach to the running exe, but that could be problematic if your code gets loaded before you have a chance to get in there.

Answer 4

I used to use the DebugBreak function for this. You could have it called conditionally based on the presence of a particular file, perhaps.

#ifdef DEBUG
if (... file exists...) {
    DebugBreak();
}
#endif

This will halt application execution until you attach a debugger or terminate the app.

Answer 5

Here's a simple solution: add a Sleep(10000); in DllMain (or some other startup code) and then use Tools / Attach to Process to attach your debugger while the code is sleeping.

Answer 6

__asm int {3};

in your DLL main. Then attach a debugger to the process? If this kills the process, then it probably has it's own int3 trap and is quitting. Are you trying to debug a copy protected game or anything like that? As they tend to do that kind of tricksy behaviour.

Answer 7

I'm thinking that there should be some call to allow the dll to spin indefinitely until a debugger is attached to the process, at which point I should be able to debug the dll by attaching to the process. Does that make sense? Is there a better way to do this?

Why not do it the way you are describing it? Just start the application that you want to debug. Attach the debugger to it, either through Visual Studio or simply by right clicking on the application in the task manager and selecting Debug. Once the debugger is attached, set a break point with F9 at suitable location in your dll code.

Answer 8

Wait until a debugger is present:

while(!IsDebuggerPresent())
{
  Sleep(0);  // yield
}

MSDN Documentation: IsDebuggerPresent().

Answer 9

Ensure that the application is indeed using the DLL that you built, in debug mode, with symbols. You can verify this by using a program such as Process Explorer (in this application, enable the lower pane in the View menu and select DLLs).

Then, in Visual Studio's Debug menu, select "Attach to Process", and select the application that uses your DLL. Your debug breakpoints should become filled in, if and when your DLL is loaded.

Answer 10

I've tried doing a 'set target application' deal, where I set the application that the dll will be called from, and the application crashes a horrible, horrible death when called that way. I don't know if the fault lies with this dll or with the executable for that behavior, and it's just one of the myriad things I'd like to solve.

Starting a process inside the debugger causes Windows to enable the NT debug heap. It sounds like the application or DLL has heap corruption or relies on the value of uninitialized heap memory.

You can disable the NT debug heap by setting the environment variable _NO_DEBUG_HEAP to 1 (on XP and later). This may make it possible to get the application to not die a horrible death when started from the debugger.

Starting the application outside the debugger will also result in the NT debug heap being disabled, and attaching a debugger later will not enable it.

Answer 11

I recently had to use one of the methods listed here: http://blogs.msdn.com/greggm/archive/2004/07/08/177418.aspx

Does that help?