C++: Loading an EXE as a DLL, local vftable problem

Question

Ok, so this one is abit long to explain so bare with me..

I have an exe named test.exe which is usually used as a stand alone application. I want to use this exe as a module (a dll) inside another application, app.exe.

The code in test.exe does something really simple like:

void doTest()
{
    MyClass *inst = new MyClass();
    inst->someMethod();
}

Where someMethod() is virtual and MyClass has a virtual d'tor.
doTest() is exported from test.exe and thus a lib called test.lib is created
app.exe is linked with this lib to statically load test.exe when it starts.

When I'm running test.exe stand-alone it runs just fine but when I'm running it loaded from within app.exe it crashes.
Stepping into the code with the debugger revealed that the crash is in the call to the virtual method. It turns out that the vftable somehow goes bad.

After some investigations it turns out that when the code inside the constructor of MyClass is running , the vftable is one thing but when the call to new returns it is replace with something else called a "local vftable". I found this obscure discussion about why this is.

After about a day of debugging it occurred to me that the pointers in this "local vftable" are the same in both cases, when test.exe is stand alone and whenit is loaded as a module. This can't be right because test.exe is loaded into a different address...
To test this theory I changed the loading address in the linker options to the one where test.exe is loaded when it is in app.exe and now, lo and behold, everything works.

Obviously, this is not a permanent solution because next time this randomly selected address may be occupied and the same problem will occur again.

So my question: Why is this "local vftable" tied to the static loading address of the exe? is loading an exe as a module a bad thing? why does the exe assume it is loaded to its static address?

Just for context: this is all done with MSVC 2008, Windows XP x64.

Answer 1

VC++ strips out reloc info from .exes by default because normally they don't need to be relocatable.

You can force it to retain the reloc info with /fixed:no. See: http://msdn.microsoft.com/en-us/library/w368ysh2.aspx

Answer 2

The workaround I ended up using is to simply add a compile configuration and compile the exe as a real dll instead of forcing it to act like one.

using /fixed:no didn't solve the problem for some reason.

Another difference I between exes and DLLs is that the entry point is different. a DLL's entry point is DllMain where as an exe has its entry point in the CRT which eventually calls main() or WinMain().