I've been looking at DynObj and decided to do my own experimentation with vftables. I'm working with Visual Studio 2010 and created a console main that instantiates an object with a virtual function that returns an std::string.
The test code in main attempts to call the object's public virtual function using a pointer obtained from the object's vftable. I've found that for a function that returns primitive type there is no problem. But when the function returns an std::string the compiler inserts an effective stack pop (add esp,4). This causes the stack check code which follows to throw an exception.
I noticed this is the norm for functions declared in global space. But the function within the class doesn't generate the ESP modifier post call.
Here's the essence of the code along with the assembly...
class VClass
{
public:
VClass() {}
~VClass() {}
virtual std::string GetString() {return "vstring";}
};
std::string StrFunc()
{
return "string";
}
void main(int argc, char* argv[])
{
VClass vClass;
__int32 ** vftabletable;
// 00BE1730 lea ecx,[ebp-18h]
// 00BE1733 call VClass::VClass (0BE1181h)
// 00BE1738 mov dword ptr [ebp-4],0
vftabletable = (__int32**)&vClass;
// 00BE173F lea eax,[ebp-18h]
// 00BE1742 mov dword ptr [ebp-24h],eax
StrFunc();
// 00BE1745 lea eax,[ebp-15Ch]
// 00BE174B push eax
// 00BE174C call StrFunc (0BE11EFh)
// 00BE1751 add esp,4
// 00BE1754 lea ecx,[ebp-15Ch]
// 00BE175A call std::basic_string<char,std::char_traits<char>,std::allocator<char> >::~basic_string<char,std::char_traits<char>,std::allocator<char> > (0BE1221h)
vClass.GetString();
// 00BE175F lea eax,[ebp-134h]
// 00BE1765 push eax
// 00BE1766 lea ecx,[ebp-18h]
// 00BE1769 call VClass::GetString (0BE1195h)
// 00BE176E lea ecx,[ebp-134h]
// 00BE1774 call std::basic_string<char,std::char_traits<char>,std::allocator<char> >::~basic_string<char,std::char_traits<char>,std::allocator<char> > (0BE1221h)
((std::string (*)())vftabletable[0][0])();
// 00BE1779 mov esi,esp
// 00BE177B lea eax,[ebp-10Ch]
// 00BE1781 push eax
// 00BE1782 mov ecx,dword ptr [ebp-24h]
// 00BE1785 mov edx,dword ptr [ecx]
// 00BE1787 mov eax,dword ptr [edx]
// 00BE1789 call eax
// 00BE178B add esp,4
// 00BE178E cmp esi,esp
// 00BE1790 call @ILT+570(__RTC_CheckEsp) (0BE123Fh)
// 00BE1795 lea ecx,[ebp-10Ch]
// 00BE179B call std::basic_string<char,std::char_traits<char>,std::allocator<char> >::~basic_string<char,std::char_traits<char>,std::allocator<char> > (0BE1221h)
}
The last call to the virtual function via the casted pointer results in:
Run-Time Check Failure #0 - The value of ESP was not properly saved across a function call. This is usually a result of calling a function declared with one calling convention with a function pointer declared with a different calling convention.
So my question is: how do I invoke the proper calling convention on a virtual function that returns a non primitive type?
Btw, when I break at 00BE178B and set next statement to 00BE178E execution completes without a problem.