ASP.NET 2.0 app runs on Win 2003 in IIS 5 isolation mode but not in (default) IIS 6 mode

Question

The app uses DLLImport to call a legacy unmanaged dll. Let's call this dll Unmanaged.dll for the sake of this question. Unmanaged.dll has dependencies on 5 other legacy dll's. All of the legacy dll's are placed in the WebApp/bin/ directory of my ASP.NET application.

When IIS is running in 5.0 isolation mode, the app works fine - calls to the legacy dll are processed without error.

When IIS is running in the default 6.0 mode, the app is able to initiate the Unmanaged.dll (InitMe()), but dies during a later call to it (ProcessString()).

I'm pulling my hair out here. I've moved the unmanaged dll's to various locations, tried all kinds of security settings and searched long and hard for a solution. Help!

Sample code:

[DllImport("Unmanaged.dll", EntryPoint="initME", CharSet=System.Runtime.InteropServices.CharSet.Ansi, CallingConvention=CallingConvention.Cdecl)]
 internal static extern int InitME();
//Calls to InitMe work fine - Unmanaged.dll initiates and writes some entries in a dedicated log file
[DllImport("Unmanaged.dll", EntryPoint="processString", CharSet=System.Runtime.InteropServices.CharSet.Ansi, CallingConvention=CallingConvention.Cdecl)]
 internal static extern int ProcessString(string inStream, int inLen, StringBuilder outStream, ref int outLen, int maxLen);
//Calls to ProcessString cause the app to crash, without leaving much of a trace that I can find so far

Answer 1

What is the error given? if the application truly crashed you might have to go into the Windows Event Log to get the stack trace of the error.

Answer 2

I've run procmon, debugdiag, tried to work with microsoft debugging tools. Each time the app crashes, Dr. Watson creates a pair of files - .dmp and .tmp (which I have tried to debug without success).

Here's the error from the Event Log:

Event Type: Error
Event Source: .NET Runtime 2.0 Error Reporting
Event Category: None
Event Code: 1000
Date: 30.09.2008
Time: 16:13:38
User: Not Applicable
Computer: APPLICATIONTEST010
Description:
Faulting application w3wp.exe, version 6.0.3790.3959, stamp 45d6968e, faulting module Unmanaged1.dll, version 0.0.0.0, stamp 48b6bfb8, debug? 0, fault address 0x00122264.

Answer 3

Back with a stack, taken from a mini dump. Seems to be a stack overflow, but I'd like to know more if someone can help me out. Results of "kb" in WinDbg with sos.dll loaded:

1beb51fc 7c947cfb 7c82202c 00000002 1beb524c ntdll!KiFastSystemCallRet
1beb5200 7c82202c 00000002 1beb524c 00000001 ntdll!NtWaitForMultipleObjects+0xc
WARNING: Stack unwind information not available. Following frames may be wrong.
1beb52a8 7c822fbe 00000002 1beb52ec 00000000 kernel32!WaitForMultipleObjectsEx+0xd2
1beb52c4 7a2e1468 00000002 1beb52ec 00000000 kernel32!WaitForMultipleObjects+0x18
1beb5308 7a2d00c4 7a0c3077 1bc4ffd8 1bc4ffd8 mscorwks!CreateHistoryReader+0x19e9d
1beb531c 7a0c312f 7a0c3077 1bc4ffd8 888d9fd9 mscorwks!CreateHistoryReader+0x8af9
1beb5350 7a106b2d 1b2733a0 00000001 1b2733a0 mscorwks!GetCompileInfo+0x345ed
1beb5378 7a105b91 1b272ff8 1b2733a0 00000001 mscorwks!GetAddrOfContractShutoffFlag+0x93a8
1beb53e0 7a105d46 1beb5388 1b272ff8 1beb5520 mscorwks!GetAddrOfContractShutoffFlag+0x840c
1beb5404 79fe29c5 00000001 00000000 00000000 mscorwks!GetAddrOfContractShutoffFlag+0x85c1
1beb5420 7c948752 1beb5504 1beef9b8 1beb5520 mscorwks!NGenCreateNGenWorker+0x4d52
1beb5444 7c948723 1beb5504 1beef9b8 1beb5520 ntdll!ExecuteHandler2+0x26
1beb54ec 7c94855e 1beb1000 1beb5520 1beb5504 ntdll!ExecuteHandler+0x24
1beb54ec 1c9f2264 1beb1000 1beb5520 1beb5504 ntdll!KiUserExceptionDispatcher+0xe
1beb57f4 1c92992d 1beb6e28 1db84d70 1db90e28 Unmanaged1!UMgetMaxSmth+0x1200ad
1beb5860 1c929cfe 00000000 1db84d70 1beb6e28 Unmanaged1!UMgetMaxSmth+0x57776
1beb58c0 1c930b04 00000000 1db84d70 1beb6e28 Unmanaged1!UMgetMaxSmth+0x57b47
1beb5924 1c99d088 00000000 1db84d70 1beb6e28 Unmanaged1!UMgetMaxSmth+0x5e94d
1beb5990 1c99c955 00000000 1beb6e28 1beb6590 Unmanaged1!UMgetMaxSmth+0xcaed1
1beb5a44 1c99e9ae 00000000 40977000 1db90e28 Unmanaged1!UMgetMaxSmth+0xca79e

Answer 4

Solution: Create a new thread in which to run the imported dll, assign more memory to its stack.

Answer 5

I think a potential problem to look for here is that your DLL could be unloaded by runtime between calls to InitME and ProcessString - so if ProcessString depends on InitME being called first, it might go "boom".

The solution to that would be using good old LoadLibrary and FreeLibrary to force runtime to keep the library loaded between calls to those two functions. GetProcAddress is not needed (as far as I can tell).