Attaching a DLL to debug?

Question

I have a DLL as an add-in in a third party application. When I launch the DLL from MS Visual Studio 2005 in debug mode the third party application will crash. Instead of launching my DLL from Visual Studio in debug mode, I try to launch the third party application first which will load my DLL. Then I attach my DLL to the third party application but the breakpoint in my DLL can't be enabled.

Has anyone been successful in debugging a DLL this way?

Answer 1

When you attach, make sure you have loaded the symbols for your DLL Also, mouse over the disabled breakpoint symbol - it has a tooltip that can help explain what the problem is with setting the breakpoint.

Common reasons are:

1. Not having correct debug symbols - right click the module in the "Modules" window and hit "Symbol load information" - this shows all the paths it was searching in for the symbol file
2. Having loaded the wrong module, or a different one than you think - check the "Modules" window to see the full path
3. Source file has changed since the module was built
4. Module not loaded (yet)

Answer 2

Making sure that the DLL is loaded before setting the breakpoints seems to work for me (this is C# however and not C++, not sure if this makes a difference). Note that the IDE could still report that the breakpoint will not be hit, although it does work - I put it down in my situation as an unmanaged - managed code interfacing quirk.

Edit: as was very kindly pointed out, this happens only in VS6 and versions prior to that. I was doing this in VS2005 purely out of habit...

Answer 3

Yes, this can be done, I do it every day. (Well, I was doing it every day in VS 2005 until we upgraded to VS 2008 a few months back, but it works the same in both versions).

You should have the project/solution in which you build your DLL loaded in Visual Studio. Then you should be able to attach to the third-party application, and Visual Studio will find the DLL loaded within it and enable your breakpoint. If the breakpoint is not enabling, the first thing I would look at is whether your DLL is actually loaded in the third party application. As someone pointed out you can see this in the Modules list in Visual Studio.

One trick that we use for situations like this is we have some code similar to this in the early startup code of our DLL:

#ifdef DEBUG
::MessageBox( NULL, "Attach debugger now", "ATTACH", MB_OK );
#endif

That way, only when debugging, you get a nice noticeable prompt that (a) your DLL did in fact load and (b) now is the time to attach the debugger. It's even fairly straightforward to build a Visual Studio macro that you can assign to a hotkey that will look for this popup in the process list and attach for you.

Warning: do NOT put this code in the DllMain() function of your DLL. It's rarely if ever safe to do anything significant in DllMain(). Instead, put it in the function that the third-party application calls first in whatever plugin API you are implementing.

If you never see the popup, then you know the third party application is actually refusing to load your DLL, or tried and failed. In this case I would look at the following:

• Make sure you are following whatever rules required by the third party plug-in API to get your DLL loaded. I.e. your DLL is in the right directory and whatever configuration settings are necessary are in place.
• If everything above is OK, it is possible LoadLibrary() is failing when the third party application tries to load you into memory. In this case the most likely culprit is you are linking to other DLLs from your code, and those aren't located along the path.

Answer 4

Thanks for replay. I should give more detail. 1) I'm using vistual studio 2005; 2) my Add-in DLL is a in-process COM.

I tried as Tim suggested to add a message box into my DLL. Starting the 3rd party App, the message box poped up, then from VS 2005 of my DLL project I atteched it to the 3rd party App.

From Modules list, I could not see my DLL is loaded and the break point is still disabled, moved mouse over the disabled break point it shown "The breakpoint will not currently be hit. No symbols have been loaded for this document".

I'm not sure why my DLL has been loaded but it won't show up in VS Modules list.

Thanks again for any help.

Answer 5

If the message box popped up, then your DLL loaded at that point. That's a good sign of course. That means you don't have any fundamental problems with getting the third party application to load your add-in.

Did the Modules list show your DLL as being unloaded after you pressed OK on the message box? That would tend to indicate that the third party application unloaded your addin after initially loading it. Perhaps you are not giving the application the right signals to tell it to keep your DLL loaded?

I would put a breakpoint immediately after the MessageBox (and breakpoints on all the other entry points that the application might call) and as soon as you attach you should be able to start single-stepping the code in your plugin. At some point you should be able to actually see when it unloads.

Answer 6

Modules list doesn't have my DLL unloaded when I pressed th OK on the message box. I did added put another break point right after the MessageBox but it just can't be activated by VS2005. If I added second MessageBox, the second box will pop up again, but the break point set at the second messagebox won't be enabled.

So the DLL has been loaded but VS2005 won't be able to debug my DLL. I checked the pdb file for my DLL, it is in the same folder where the DLL is.

Any suggestions?

Thanks again for you time.

Answer 7

Use a DebugBreak to attach to the debugger.

DebugBreak:

This function causes a breakpoint exception to occur in the current process so that the calling thread can signal the debugger and force it to take action.

Answer 8

The guy from 3rd part Application told me their trick, and it worked. What I did is the following: 1) open VS2005 and open my DLL project 2) start the 3rd party App 3) start Windows Task Manager, find and select the 3rd party App then right click mouse select Debug from the pop up menu 4) select Yes from "Task Manager Warning" dialog 5) select my DLL project from "Visual Studio Just-In-Time Debugger", then click Yes. 6) this starts my DLL from VS2005 and all the break points are enabled