Run-Time Check Failure #0 vb.net callback from C dll

Question

Hi all. I'm writing Add-inn Application A in VB.Net and DLL B in C language. Application A pass callback method to dll B. When certain event occur the dll invoke the callback from A. Whole works fine on my PC but when I move it to Notebook I get an error:

Run-Time Check Failure #0 - The value of ESP was not properly saved across a function call. This is usually a result of calling a function declared with one calling convention with a function pointer declared with a different calling convention.

This is part of C code:

typedef void (__cdecl * OFFICE_PTR)();
void  TAPIClient::tapiCallBack(
DWORD hDevice,
DWORD dwMessage,
DWORD dwInstance,
DWORD dwParam1,
DWORD dwParam2,
DWORD dwParam3){
switch (dwMessage)
{
 case LINE_CALLSTATE:
  switch (dwParam1)
  {
   case LINECALLSTATE_OFFERING:
         if(dwInstance!=NULL)
    {
     try
     { 
      OFFICE_PTR vbFunc =(OFFICE_PTR)dwInstance;
      vbFunc( );//Critical moment
     }
     catch(...)
     {
      MessageBox (NULL, L"( (OFFICE_PTR)dwInstance )(&sCallNr)",L"ERROR",MB_OK);
     }
    }
   break;
  };
 break;
}

}

Where dwInstance is a address of application A callback method

This is part of VB.Net code:

Public Class TapiPlugin

Public Delegate Sub P_Fun()

Private Declare Function startSpy _
    Lib "TAPIClient.dll" _
    (ByVal pFun As P_Fun) As IntPtr

Public Shared Sub simpleTest()
    MsgBox("Plugin sub simpleTest")
End Sub

Public Sub onStart()
    Dim pBSTR As IntPtr
    pBSTR = startSpy(AddressOf simpleTest)
    MsgBox(Marshal.PtrToStringAuto(pBSTR))
    Marshal.FreeBSTR(pBSTR)
End Sub

End Class

The Error occur when I try call 'vbFunc( )'. I would be grateful for any help. :D

Answer 1

Do the two computers have different pointer sizes perhaps? Maybe one is a 64 bit machine and the other only 32?

typedef void (__cdecl * OFFICE_PTR)();
void  TAPIClient::tapiCallBack(
DWORD hDevice,
DWORD dwMessage,
DWORD dwInstance,
...){
...
        OFFICE_PTR vbFunc =(OFFICE_PTR)dwInstance;
        vbFunc( );//Critical moment

The DWORD type is not really valid for passing pointer types. You should be using INT_PTR I guess.

Answer 2

I thing it is not a reason to check it out I passed the callback as global pointer of type OFFICE_PTR and i get the same result. On PC it work fine on Notebook it crash :(

A have to apologies for a mistake I wrote that the def look like:

typedef void (__cdecl * OFFICE_PTR)();

but for real it looks like

typedef void (__stdcall * OFFICE_PTR)();

Answer 3

If the calling convention is cdecl, then you need to declare your delegate like this:

<UnmanagedFunctionPointer(CallingConvention.Cdecl)>
Public Delegate Sub P_Fun()

You can only do this in .NET 2.0 and after, as the attribute was not introduced before then (and the interop layer was not changed to acknowledge it before that).

If the calling convention is indeed stdcall then the delegate can remain as is. You said it is stdcall, but I have doubts, since the exception is explicitly telling you that there might be a mismatch in calling conventions.