Stack issues when calling a DLL compiled with Visual C++ in GCC

Question

I'm trying to call some functions in a DLL compiled with (I believe) Visual C++ from my program, which is compiled using GCC.

To call the functions in the DLL, I do a LoadLibrary() on the DLL, and then a GetProcAddress() to get the address of a particular function, which I then call. This function returns a list of pointers to the functions in the DLL I'm to call.

Well, when I try to call those functions, they don't work properly. I ran my program through a debugger, and it looks like the DLL library function is looking for one of the passed arguments at ebp+8, even though GCC has put it at ebp-24.

It looks definitely like a stack issue. What's more, when the GCC program function which calls the DLL function returns, my program crashes -- so something screwey is going on with the stack. Does anyone know what I need to do in order to fix this? I'm not able to access the DLL code.

Also: I tried putting __cdecl and __stdcall before the DLL function definition in my program's source file, but this changes nothing.

Answer 1

Looks like a calling convention problem. Make sure you're putting the calling convention tag in the right place. With GCC, it should look like this:

typedef int (__stdcall *MyFunctionType)(int arg1, const char *arg2);
MyFunctionType myFunction = (MyFunctionType)GetProcAddress(myModule, "MyFunction");
// check for errors...
int x = myFunction(3, "hello, world!");

[EDIT]

Looks like your problem has nothing to do with calling conventions (although getting them right is important). You're misusing BSTRs -- a BSTR is not a simple char* pointer. It's a pointer to a Unicode string (wchar_t*), and furthermore, there is a 4-byte length prefix hidden before the first characters of the string. See MSDN for full details. So, the call to SetLicense() should look like this:

BSTR User = SysAllocString(L"");  // Not sure if you can use the same object here,
BSTR Key = SysAllocString(L"");   // that depends on if SetLicense() modifies its
                                  // arguments; using separate objects to be safe
// check for errors, although it's pretty unlikely
(textCapLib.sdk)->lpVtbl->SetLicense((textCapLib.sdk), User, Key);
SysFreeString(User);  // Hopefully the SDK doesn't hang on to pointers to these
SysFreeString(Key);   // strings; if it does, you may have to wait until later to
                      // free them