Anyone encountered a INT 3 debugger break in mscorwks.dll?

Question

We host the .NET runtime as part of a Win32 program, and lately it has begun to consistently break at a specific address, in mscorwks.dll.

At the specified address, there is a 0xCC byte, which is a INT 3 instruction, which fires the debugger.

Has anyone else seen this?

I can't see enough information in the dll to know specifically where it is, function or source-wise, but it is definitely not in any of our libraries.

The call-stack looks like this (this is from Delphi 2007):

:7a04f02a ; C:\WINDOWS\Microsoft.NET\Framework\v2.0.50727\mscorwks.dll
:7a052fc6 ; C:\WINDOWS\Microsoft.NET\Framework\v2.0.50727\mscorwks.dll
:7a053e72 ; C:\WINDOWS\Microsoft.NET\Framework\v2.0.50727\mscorwks.dll
:7a03b970 ; C:\WINDOWS\Microsoft.NET\Framework\v2.0.50727\mscorwks.dll
:7a00351e ; C:\WINDOWS\Microsoft.NET\Framework\v2.0.50727\mscorwks.dll
:7a0255e0 ; C:\WINDOWS\Microsoft.NET\Framework\v2.0.50727\mscorwks.dll
:79e71e6d ; C:\WINDOWS\Microsoft.NET\Framework\v2.0.50727\mscorwks.dll
:7e42b372 USER32.MoveWindow + 0xd4
:7e4565b7 USER32.GetRawInputDeviceInfoW + 0x5f
:7e42ce7c USER32.SetLayeredWindowAttributes + 0x6a

The address it fails on is 0x7A04F029, disassembly in memory looks like this:

7A04F020 call $7a14be35
7A04F025 cmp [esi],ebx
7A04F027 jz $7a04f02a
7A04F029 int 3                      <-- here's the culprit
7A04F02A mov byte ptr [ebp-$04],$02
7A04F02E lea ecx,[ebp-$00000224]
7A04F034 call $79e82214

It's easy enough to get rid of, just patch that byte in memory with a 0x90 (NOP) each time, but the next time we fire the debugger, the dll is of course reloaded.

I've got half a mind to try to patch the file itself, but I'm not sure that's a good idea either. I've verified that the CC byte is part of the dll, found the surrounding byte pattern with a hex editor.

Any tips on how to fix this? Anyone else seen this?

Solved

As mentioned in the answers, this was a LoaderLock problem. A DLL that was built in Win32 needed to expose a couple of functions for our .NET system, in a way not tied through the way we host the .NET runtime. The project included a couple of units that contained the functions that was going to be exposed, but unfortunately also added lots and lots of code that was useful in an application, but not for this DLL.

Separating out the functions I needed reduced the size of the DLL from 7MB to around 100KB and got rid of the LoaderLock as well.

Answer 1

There's a discussion about a similar problem here. The most interesting part is this:

Mscorwks.dll is just the messenger. It is telling you that it detected a loaderlock. It is your code that caused it.

Answer 2

Just a friendly reminder, NOP'ing the instruction might not be such a good idea since there's probably a reason you're ending up at that breakpoint. But I'm sure you already realized that :)

Answer 3

You should be able to get the symbols for mscorwks - I have a couple of versions here, so MS symserv has definitely served them in the past.

That might help understand what's happening a bit more.

From the fact that you have trace in USER32.dll, I suspect you already have some symbol stuff configured, but if you don't, then this might help: http://www.microsoft.com/whdc/devtools/debugging/debugstart.mspx

I don't use Delphi myself, but I'd guess it uses dbghelp.dll to do symbol/PDB stuff, so it can probably be set up use symserv even if it doesn't have any explicit UI to help.