views:

8

answers:

0

I've been looking at DynObj and decided to do my own experimentation with vftables. I'm working with Visual Studio 2010 and created a console main that instantiates an object with a virtual function that returns an std::string.

The test code in main attempts to call the object's public virtual function using a pointer obtained from the object's vftable. I've found that for a function that returns primitive type there is no problem. But when the function returns an std::string the compiler inserts an effective stack pop (add esp,4). This causes the stack check code which follows to throw an exception.

I noticed this is the norm for functions declared in global space. But the function within the class doesn't generate the ESP modifier post call.

Here's the essence of the code along with the assembly...

class VClass
  {
  public:
    VClass() {}
    ~VClass() {}

    virtual std::string GetString() {return "vstring";}
  };

std::string StrFunc()
  {
  return "string";
  }

void main(int argc, char* argv[])
  {
  VClass vClass;
  __int32 ** vftabletable;
    // 00BE1730  lea         ecx,[ebp-18h]  
    // 00BE1733  call        VClass::VClass (0BE1181h)  
    // 00BE1738  mov         dword ptr [ebp-4],0  

  vftabletable = (__int32**)&vClass;
    // 00BE173F  lea         eax,[ebp-18h]  
    // 00BE1742  mov         dword ptr [ebp-24h],eax  

  StrFunc();
    // 00BE1745  lea         eax,[ebp-15Ch]  
    // 00BE174B  push        eax  
    // 00BE174C  call        StrFunc (0BE11EFh)  
    // 00BE1751  add         esp,4  
    // 00BE1754  lea         ecx,[ebp-15Ch]  
    // 00BE175A  call        std::basic_string<char,std::char_traits<char>,std::allocator<char> >::~basic_string<char,std::char_traits<char>,std::allocator<char> > (0BE1221h)

  vClass.GetString();
    // 00BE175F  lea         eax,[ebp-134h]  
    // 00BE1765  push        eax  
    // 00BE1766  lea         ecx,[ebp-18h]  
    // 00BE1769  call        VClass::GetString (0BE1195h)  
    // 00BE176E  lea         ecx,[ebp-134h]  
    // 00BE1774  call        std::basic_string<char,std::char_traits<char>,std::allocator<char> >::~basic_string<char,std::char_traits<char>,std::allocator<char> > (0BE1221h)  

  ((std::string (*)())vftabletable[0][0])();
    // 00BE1779  mov         esi,esp  
    // 00BE177B  lea         eax,[ebp-10Ch]  
    // 00BE1781  push        eax  
    // 00BE1782  mov         ecx,dword ptr [ebp-24h]  
    // 00BE1785  mov         edx,dword ptr [ecx]  
    // 00BE1787  mov         eax,dword ptr [edx]  
    // 00BE1789  call        eax  
    // 00BE178B  add         esp,4  
    // 00BE178E  cmp         esi,esp  
    // 00BE1790  call        @ILT+570(__RTC_CheckEsp) (0BE123Fh)  
    // 00BE1795  lea         ecx,[ebp-10Ch]  
    // 00BE179B  call        std::basic_string<char,std::char_traits<char>,std::allocator<char> >::~basic_string<char,std::char_traits<char>,std::allocator<char> > (0BE1221h)  

  }

The last call to the virtual function via the casted pointer results in:

Run-Time Check Failure #0 - The value of ESP was not properly saved across a function call. This is usually a result of calling a function declared with one calling convention with a function pointer declared with a different calling convention.

So my question is: how do I invoke the proper calling convention on a virtual function that returns a non primitive type?

Btw, when I break at 00BE178B and set next statement to 00BE178E execution completes without a problem.