RtlValidateHeap Problem

Question

I have a Windows DLL that at one point, returns a pointer to a class that is new'ed in the DLL's code. The class itself is a very thin wrapper around another class private to the DLL.

The calling executable has no problem working with this class, everything works fine, with the exception that when the calling executable tries to delete this class, I get a RtlValidateHeap error.

The reasons behind the error makes sense, the exe is trying to free memory on the DLL's heap, which is normally a bad thing.

I've come up with a few possible solutions:

• Override the class's new operator to allocate its memory off of the executable's heap (provided I can even get at that heap space). The wrapper is very thin, so I would only be allocating a few bytes from the exe's heap.
• Provide a special destruction function for this class (yuck).
• Tell the user not to destroy the class and live with leaks (no way!)

Is there a 'normal' way of doing this?

Answer 1

I've seen this behaviour is when one of the binaries was built with debug options (which uses a debug heap that, amongst other things, uses RtlValidateHelp) and the other binary was built as a release binary. It might be worth checking that you only use like with like...

Answer 2

Most libraries that use this sort of functionality, whether C or C++, actually go with your second option - specific functions to get and release objects. It allows distribution of binary DLLs that don't have to be compiled identically to the calling executable.

Answer 3

Are you loading the DLL dynamically at runtime (by calling LoadLibrary), or are you linking to using a .lib file during the link phase of your exe? If you're dynamically loading/unloading the DLL then when the delete happens is the DLL still loaded? This is a classic problem for plug-in type designs and the typical solution is something like your second proposal - specific functions to create and delete the objects in question.

Answer 4

Option 2 is pretty normal, eg, a Release() method that just does a "delete this".

Note that normally the DLL doesn't have it's own heap as such - most DLLs will share the process heap (see GetProcessHeap), but often the language runtime will decorate or add padding around allocated memory - so the pointer your dll ends up with is not the same pointer the heap will expect in a call to free.

The general rule is always release allocated memory with the matching free function, from within the same module.

There are some exceptions though - memory you know has been allocated directly with HeapAlloc from a known heap can be freed from another module. GlobalAlloc/GlobalFree works across modules. Strings allocated with SysAllocString can be freed from another module. Memory you know has been allocated by the same DLL version of the runtime can be freed from another module, but I'd stay away from this.

Answer 5

By far the easiest solution is (for the Visual Studio users) to use /MD or /MDd everywhere. If the CRT itself is in a DLL, there's only one heap. Your app and your DLL will share it.

Answer 6

On the DLL side, I would use something along these lines:

dll.h:  /* all exported, included by dll user */
class dll_class {
  public:
    virtual void foo() = 0;
}
dll_class* create_class();
void delete_class(dll_class*);

dll.cpp: /* implement the creator and destroyer */
dll_class* create_class() { return new dll_class_imp(); }
void delete_class(dll_class* pClass) { delete pClass; }

dll_imp.h: /* implement the actual dll_class logic */
class dll_class_imp : public dll_class {
  public:
    virtual void foo() {...}
}

Now, on the DLL user's side, dll.h will be included, and used like so:

dll_class* pClass = create_class();
pClass->foo();
delete_class(pClass);

this way, there's a clear distinction between the DLL's part and the user's part. Being an abstract class, the user cannot create a dll_class using new; and it only makes sense you delete object similarly to the way you've created them, i.e. with the exported functions.

Answer 7

What looks like will work for us was to override the new and delete operators for the problem class. It turned out that there were 3 situations that could happen:

• EXE new's object, EXE delete's object
• DLL new's object, DLL delete's object
• DLL new's object, EXE delete's object

The new and delete operators ensure that the DLL would be allocating and freeing all memory allocations. So far, so good - testing shows that the operators are always being used and the memory is coming from the DLL's heap.