Avoiding stack overflows in wrapper DLLs

Question

I have a program to which I'm adding fullscreen post-processing effects. I do not have the source for the program (it's proprietary, although a developer did send me a copy of the debug symbols, .map format). I have the code for the effects written and working, no problems.

My issue now is linking the two.

I've tried two methods so far:

Use Detours to modify the original program's import table. This works great and is guaranteed to be stable, but the user's I've talked to aren't comfortable with it, it requires installation (beyond extracting an archive), and there's some question if patching the program with Detours is valid under the terms of the EULA. So, that option is out.

The other option is the traditional DLL-replacement. I've wrapped OpenGL (opengl32.dll), and I need the program to load my DLL instead of the system copy (just drop it in the program folder with the right name, that's easy).

I then need my DLL to load the Cg framework and runtime (which relies on OpenGL) and a few other things. When Cg loads, it calls some of my functions, which call Cg functions, and I tend to get stack overflows and infinite loops. I need to be able to either include the Cg DLLs in a subdirectory and still use their functions (not sure if it's possible to have my DLLs import table point to a DLL in a subdirectory) or I need to dynamically link them (which I'd rather not do, just to simplify the build process), something to force them to refer to the system's file (not my custom replacement).

The entire chain is: Program loads DLL A (named opengl32.dll). DLL A loads Cg.dll and dynamically links (GetProcAddress) to sysdir/opengl32.dll. I now need Cg.dll to also refer to sysdir/opengl32.dll, not DLL A.

How would this be done? Edit: How would this be done easily without using GetProcAddress? If nothing else works, I'm willing to fall back to that, but I'd rather not if at all possible.

Edit2: I just stumbled across the function SetDllDirectory in the MSDN docs (on a totally unrelated search). At first glance, that looks like what I need. Is that right, or am I misjudging? (off to test it now)

Edit3: I've solved this problem by doing thing a bit differently. Instead of dropping an OpenGL32.dll, I've renamed my DLL to DInput.dll. Not only does it have the advantage of having to export one function instead of well over 120 (for the program, Cg, and GLEW), I don't have to worry about functions running back in (I can link to OpenGL as usual). To get into the calls I need to intercept, I'm using Detours. All in all, it works much better. This question, though, is still an interesting problem (and hopefully will be useful for anyone else trying to do crazy things in the future). Both the answers are good, so I'm not sure yet which to pick...

Answer 1

SetDllDirectory probably won't work. Cg.dll likely just links to OpenGL.dll. When the OS loads Cg.dll, it sees that there's already a module loaded with that name (yours), so it links Cg with that instead of going off to find some other copy. That is, the search order that SetDllDirectory modifies never even comes into play because the OS doesn't do any searching.

I suspect your best bet will indeed be to detect re-entrant calls to your library. When you detect one, instead of doing your custom handling, forward the call directly to the real OpenGL library, which you have a reference to due to calling LoadLibrary and then GetProcAddress for each of the library's functions.

Answer 2

You can use the magic of activation contexts to attempt to solve your problem.

A lot hinges on weather the 3rd party components in your system already have manifests - and how much tampering with those manifests might constitute a license violation.

In order to resolve dll versioning issues, Windows XP got a technology called activation contexts. Sometimes known as side-by-side assemblies, or even something horrible like Application Isolation

To summarize lots of reading into a small space: manifests are chunks of XML data that can describe an assembly, or describe a dependency on assemblies. An assembly is a manifest, plus its dll.

The reason this exists is so that an assembly can take a simple dll. "comctl32.dll" and its version number (v6), and create a thing with a bigger more unique name, such that multiple versions of the simple dll can be installed in the same place safely. Assemblies are intended to be installed in C:\Windows\WinSxS.

When a manifest file describes the dlls in an assembly, its called an assembly manifest. And usually has a different name to the dll.

When a manifest file describes the assemblies a dll or exe uses, its called an application manifest, and is usually embedded as an RT_MANIFEST resource - in EXEs with a res id of 1, in Dlls with a res id of 2 - or on disk as a file with a name "appname.exe.manifest" / "dllname.dll.2.manifest". Application manifests define a thing called an activation context - which is basically a namespace that windows will search in for things. Each manifest creates an activation context. Each activation context has a mapping of simple dll names to assemblies.

So, if you create a assembly with your opengl32.dll file, and create an activation context for the app.exe referencing (the local opengl32.dll) file, then, potentially, all the remaining dlls can (and will) continue to use the systems opengl32.dll file despite the fact that the names are very similar coff.

The problem is, the res-id of the application manifest - 1 - means that its used to create the process default activation context - so ALL dlls that do not have their own explicit manifests (Cg?) are going to search the process default space and find that opengl32.dll

So you have to create manifests for every dll that doesn't already embed one, making sure to simply NOT reference your opengl32.dll assembly, which should allow then to revert to the default search order and find it in the normal system32 location.

this means that your opengl32.dll cannot be in the exe's folder as that folder is searched for dll's before system32 (the fact you rely upon for hooking).

We are saved by the rather simple search order the system takes when searching for assemblies. first it searches in WinSxS. Your Opengl32.dll won't be in there, installing there is a hard problem. Then it searches in the exe's folder, for a subfolder with the name of the assembly, THEN it searches in the exe's folder for the assemblies manifest directly.

This means you can create an assembly - called something like: "OpenGLHook" And your folder structure would look like:

\appfolder\
  app.exe
  app.exe.manifest                  - contains a dependentAssembly node to OpenGLHook
  OpenGLHook\OpenGLHook.manifest    - contains a file name=opengl32.dll
  OpenGLHook\opengl32.dll           - your hook dll
  yourimpl.dll                      - your implementation dll that linkgs to cg.dll
  cg.dll                            - cg libraries
  cg.dll.2.manifest                 - a stub manifest you put together to ensure cg
                                      doesnt use the app default activation ctx.

Um, Good luck?