How to protect yourself against shell DLLs loaded into your process?

Question

When you use a standard Windows "file open" dialog using GetOpenFileName(), the shell will load various DLLs that it requires to display the file list, including custom ones.

In my application, I found that the DLL that TortoiseCVS uses to draw overlays on the icons was calling GdiPlusShutdown(), and so some time after displaying a "file open" dialog, the TortoiseCVS DLL would be unloaded, it would shut down GDI+ and my graphics functions would all fail!

It seems quite bad that basically any old DLL could be loaded by my application at any time and start doing random things to its state. The workaround in my case was quite simple - just restart GDI+ if I detect that it's been shut down. However had this happened on a client's machine where I couldn't debug it, it would have been a lot more challenging to figure out what was going on.

Can anybody offer any insight? What could I do to stop this from happening?

Answer 1

I've had to deal with the crap that Dell puts on its machines, in particular wxVault. My solution was to "simply" patch the code. Slightly tricky with DEP, but still doable. You could have a peek at Microsoft Detours, which is a slightly more structured way to do the same. You'd still have the DLL load, but at least you can stop it calling functions that it should not be calling.

At to why Windows has such a crappy mechanism, read Raymond Chen's "Old New Thing" blog or book.