weird stack corruption due to a dll call

Question

I'm trying to make a call to a DLL function (via GetProcAddress etc) from C, using lcc compiler. The function gets called and everything goes well, but it looks like the top of the stack gets corrupted. I've tried to play with calling conventions (__stdcall / __cdecl), but that didn't help.

Unfortunately I don't have access to the dll code, and have to use the lcc compiler.

I found that this simple hack avoids stack corruption:

void foo(params)
{
  int dummy;
  dll_foo(params);
}

Here dll_foo is the pointer returned by GetProcAddress, and the stack is kind of protected by the dummy variable. So it's not the stack pointer that gets corrupted, but the data at the top of the stack. It works like this, but I'd like to know the reason of the corruption.

Any ideas?

UPD: As asked in the comments, here are the actual function types:

typedef unsigned char (CALLBACK Tfr)(unsigned char);
typedef void (CALLBACK Tfw)(unsigned char,unsigned char);
typedef int (CALLBACK Tfs)(int);
typedef void (CALLBACK Tfwf)(int*,int);

All they show a similar behavior.

Unfortunately, it is not so straightforward to attach a debugger, as the code is compiled and launched by Matlab, using the LCC compiler, and there is no debugging support. Probably I will have to reproduce this problem in a standalone configuration, but it is not that easy to make it.

Answer 1

It sounds to me like you were on the right track with looking at the calling convention. The main thing you need to do is ensure that the caller and callee are both using the same convention. Typically for a DLL, you want to use __stdcall for both, but if (as you say) you have no control over the DLL, then you need to modify your code to match what it's doing. Unfortunately, it's almost impossible to guess what that is -- I'm pretty sure lcc (like most C and C++ compilers) can produce code to use a variety of conventions.

Based on your hack working by putting an extra dword on the stack, it sounds like you currently have a mismatch where both the caller and the callee are trying to clear arguments off the stack (e.g., the caller using __cdecl and the callee using __stdcall.

Answer 2

Sounds like you use MSVC, Debug + Windows + Registers. Look at the value of ESP before and after the call. If it doesn't match then first change the calling convention in the function pointer declaration (did you do that right?) If it still doesn't match then it is __stdcall and you haven't guessed the arguments you need to pass correctly.

Or the function could just clobbers the stack frame, it isn't impossible.

Posting your function pointer declaration that shows the real arguments would probably help diagnose this better.

Answer 3

You could try to "follow" the call to dll_foo() i assembler using a debugger, at check out exactly what the routine does stack-wise.