How to detect if an OpenGL debugger is being used?

Question

Is there any way of detecting from my Windows OpenGL application if a debugger (such as gDEBugger) is being used to catch OpenGL calls? I'd like to be able to detect this and terminate my application if a debugger is found, to avoid shader code and textures from being ripped. The application is developed in C++ Builder 6.

Answer 1

Even if you could find a way to do this it would be a futile attempt because the shader source can be asked for by simply calling glGetShaderSource() at any moment.

An external process can inject a thread into your process using CreateRemoteThread() and then copy back the result with ReadProcessMemory(). This process can be made really simple (just a couple of lines) with the detours library by microsoft.

Alternatively, if creating a remote thread is too much of a hassle, a disassembler such as Ollydgb can be used to inject the a piece of code into the normal execution path which simply saves the shader code into a file just before it is invoked.

Finally, The text of the shader needs to be somewhere in your executable before you activate it and it can probably be extracted just by using a static inspection of the executable with a tool like IDAPro. It really doesn't matter if you encrypt it or compress it or whatever, if its there at some point and the program can extract it then so can a determined enough cracker. You will never win.

Answer 2

I'm not 100% sure but I believe that Graphic Remedy replace the Windows opengl32 dll with their own opengl32.dll file for hooking gl calls.

So if it is the case, you just have to check the dll version and terminate if it's not what you expect.

Answer 3

Overall, there is no way to detect each and every such "debugger". A custom OpenGL32.dll (or equivalent for the platform) can always be written; and if there is no other way, a virtual graphics card can be designed specifically for purposes of ripping your textures.

However, Graphic Remedy does have some APIs for debugging provided as custom OpenGL commands. They are provided as OpenGL extensions; so, if GetProcAddress() returns NULL on those function calls, you can be reasonably sure it's not gDEBugger. However, there are already several debuggers out there, and, as already mentioned, it's trivial to write one specifically designed for ripping out resources.

Perhaps the closest you can get is load C:\windows\system32\opengl32.dll directly, however that can break your game horribly on future releases of Windows so I'd advise against it. (And this still wouldn't protect you against those enterprising enough to replace system-wide OpenGL32.dll, or who are perhaps using other operating systems).