Incorrect vtable layout for class exported by DLL: request for clarification regarding headers and vtable construction.

Question

Hello all,

Although the problem at hand is solved, it has me a little confused as to what data is used to construct the vtables for a class and where the layout for the vtable is stored. If anyone can provide clarification or point me towards some information which might satiate my curiosity, I would greatly appreciate it.

Background

1. Two separate VC6.0 projects: one for an exe, one for a dll.
2. Application contains .lib, .dll and .h files from dll project releases.
3. When a new release is ready, the .lib, .dll, and .h files are copied into the exe project from the dll project.
4. I have inherited this scheme.

Problem:

I recently made a changes to the DLL and copied over the .lib and .dll but forgot to copy the header files. There had been some hierarchy changes and, as a result, the vtable for one particular class (call it InternalClass) had changed.

The exe does not directly invoke any methods on InternalClass. Instead, it creates an instance of a different class (call it InterfaceClass) which encapsulates a pointer to an InternalClass object and invokes various methods against that pointer.

At runtime, calls made from within InterfaceClass methods to InternalClass methods were actually invoking the wrong methods (i.e. InterfaceClass would invoke InternalClass::A and InternalClass::B would actually run). Looking at the asm, it turns out that the fixup or thunk (I'm sorry if this is the wrong jargon!) was using the correct offset into the vtable. However, the vtable itself contained the list of pointers you would expect from the old header files.

Actual Question:

I realized my mistake, copied the header files over, recompiled and all was well. However, I am left with one nagging question:

When is the layout for the vtables determined and what information is used to construct the vtable for these classes at runtime? That is, it seems to me that the proper vtable must have been assembled when the dll was compiled so that the offsets, etc could be used for the calls from InterfaceClass to InternalClass. Why, then, was an outdated vtable used at runtime? Is the layout also determined separately when the exe is compiled using the headers it has?

I'm not sure if this is at all clear. Let me know if what I'm asking is too convoluted. Thanks!

Answer 1

Hmm, heavy duty implementation details of a 12 year old compiler. When you use __declspec(dllexport) on a class, the linker exports the members of the class. Not the vtable. That gets reconstructed in the client of the class when the compiler parses the class declaration in the header file. The linker fills in that local vtable with the exported members addresses.

Would it work better if the compiler simply exported the vtable from the DLL? Maybe, but exporting such implementation details is awfully brittle. The fear of painting yourself into a corner that you cannot get out of because of backwards compatibility is a strong motivator, I imagine.

Answer 2

The way your describe the interaction of those two classes isn't totally clear on one very important detail: Is your so called InternalClass being exported from the dll or not? Or to put the question a different way: Does the code in those copied header files use pointers to InternalClass or is all knowledge of that class hidden solely within the dll?

If that class in question actually is completely hidden within the dll, then I'm not sure how this problem could happen. But from what you describe, it really sounds as if InternalClass is being exposed by the dll.

If those headers that get copied over contain code similar in concept to this:

class InterfaceClass
{
    InternalClass* Internal;

    void DoSomething()
    {
        Internal->DoSomething();
    }
};

Then the InternalClass must be exposed, otherwise the exe's project would fail to compile the header. (So it's not really internal after all.)

And if that is the case, then it easily explains your problem. The exe is being built against a different declaration of InternalClass than the dll was built with. That means each compiles their own version of the virtual table, and those versions are different.

Then at runtime, the exe code is handed an object created from within the dll, and thus with its vtable pointer pointing to the dll's version of the vtable. The exe code then tries to work through that vtable pointer as if it pointed to the exe's version of the vtable. And obviously that causes problems.