Stack overflow from native DLL called from managed application

Question

I'm getting the infamous 0xC00000FD (stack overflow) exception from my application when I call into a native DLL. This is being done from a managed C# application on Windows CE 5 (SH4 processor). Compiling the same DLL for Windows XP using the same managed application and everything works fine (no overflow). The routine in the DLL is doing some very complex recursion which is ultimately what is causing the overflow, but again, it works fine on a PC.

It seems like I may just need to adjust the stack size when building the DLL? I believe the default stack size for both CE and XP are both 1MB when using the Visual C compiler (I'm using Visual Studio 2005, if that matters). If they both default to the same size, I'm not sure why one would overflow and the other would not, however. I tried adjusting the stack size with the /F compiler flag and the /STACK linker flag but that didn't seem to do anything. It is also not entirely clear to me that I can specify the stack size in a DLL, but rather the executable must set it. But if that is the case how would I adjust the stack size for my managed process to use when calling into the native DLL?

Answer 1

Here's a link to a discussion of the memory architecture of Windows CE.

Ultimately, there is very little you can do to adjust the stack size programmatically or at compile time; the OS will vary it based on a number of factors, including memory availability. Setting stack size with /F only sets a default value that can (and often will) be ignored by the OS. And /F doesn't work on a .DLL, anyway; it only works when compiling the executable. .DLLs don't have a stack; that belongs to the thread and (ultimately) to the process.

It might be time to go back to the drawing board to remove your recursion.