Is it possible to replace the global "operator new()" everywhere?

Question

I would like to replace the global operator new() and operator delete() (along with all of their variants) in order to do some memory management tricks. I would like all code in my application to use the custom operators (including code in my own DLLs as well as third-party DLLs). I have read things to the effect that the linker will choose the first definition it sees when linking (e.g., if the library that contains your custom operator new() is linked first, it will "beat" the link with the CRT). Is there some way to guarantee that this will happen? What are the rules for this, since this really is a multiply-defined symbol (e.g., void* operator new(size_t size) has two definitions in the global namespace)?

What about third-party DLLs that may be statically linked with the CRT? Even if they are dynamically linked with the CRT, is there some way I can get them to link with my operator new()?

Answer 1

EDIT: After re-reading the question, I realize that your emphasis is on third party components. So in that case the answer is no. There are way too many variables involved.

---

(Referring to replacing "new" in your own code) The short answer is yes. This is a common trick in systems that do their own memory management. As you may have guessed, it's a very tricky thing to get right, and can easily blow up in your face if you're not careful, but it's certainly plausible and is used by lots of software. (I recall reading through some code that does this in the Doom 3 SDK, for example)

As for replacing new and delete in a 3rd party DLL, that's not going to happen (at least not safely). For one, how do you know that they even use new and delete? Not to mention they may be doing their own variant of this technique and relying on some side effect of it to function properly. And even if you get it working now, how do you know that some future version of that component wont break it? Feel free to muck with your own memory all you want, but for the sake of you and your users sanity, please don't try to micro-manage the memory of other binaries.

Answer 2

The C++ standard explicitly allows you to write your own global operator new and delete (and array variants). The linker has to make it work, though exactly how is up to the implementors (e.g., things like weak externals can be helpful for supplying something if and only if one isn't already present).

As far as DLLs go, it's going to be tricky: a statically linked DLL clearly won't use your code without a lot of extra work. Static linking means it already has a copy of the library code copied into the DLL, and any code in the DLL that used it has the address of that code already encoded. To get around that, you'd have to figure out where the code for new is in the DLL and dynamically patch all the code that calls it to call yours instead).

If the DLL links to the standard library dynamically, it gets only marginally easier -- the import table still encodes the name of the DLL and function in that DLL that provides what it needs. That can be gotten around (e.g. with something like Microsoft's Detours library) but it's somewhat non-trivial (though certainly easier than when the DLL links the standard library statically).