Access violations in strange places when using Windows file dialogs

Question

A long time ago I found out that I was getting access violations in my code due to the use of the Delphi Open File and/or Save File dialogs, which encapsulate the Windows dialogs. I asked some questions on a few forums and I was told that it may have been due to the way some programs add hooks to the shell system that result in DLLs getting injected in every process, some of which can cause havoc with a program. For the record, the programming environment I use is Delphi 6 Professional running on Windows XP 32-bit.

At the time I got around it by not using Delphi's Dialog components and instead calling straight into comdlg32.dll. This solved the problem wonderfully.

Today I was working with memory mapped files for the first time and sure enough, access violations started cropping up in weird parts of the code. I tried my comdlg32.dll direct calls and this time it didn't help. To isolate the problem as a test I created a list box with the exact same files I was using during testing. These are the exact same test files I was selecting from an Open File dialog and then launching my memory mapped file with. I set things up so that by clicking on a file in the list box, I would use that file in my memory mapped file test instead of calling into a comdlg32.dll dialog function to select a test file.

Again, the access violatons vanished. To show you how dramatic a fix it was I went from experiencing an access violation within 1 to 3 trials to none at all. Unfortunately, it's going to bite me later on of course when I do need to use file dialogs.

Has anyone else dealt with this issue too and found the real culprit? Did any of you find a solution I could use to fix this problem instead of dancing around it like I am now?

Thanks in advance.

Answer 1

I don't understand how not using the Delphi dialog components avoids shell extension DLL's causing havoc in your program if you are then calling into COMDLG32.DLL directly. You are still using the common dialogs, and those shell extensions are still being injected.

More likely is that not using the components had a side effect in your code that obfuscated or masked the underlying problem, mitigating it to the point where it appeared to have been resolved.

I suspect that is the case here too.

I don't know what sort of hokey shell extensions you have installed on your system - perhaps you have some odd-ball extension that is causing some problems. All I can say is that in 15 odd years of Win32 programming with Delphi I have never known or even heard of the Delphi common file dialog components to be responsible for such behaviour.

A simple way to test this of course would be to take the compiled EXE that exhibits the access violations on your machine, and run the same EXE on some other "clean-room" XP machine, i.e. with no 3rd party shell extensions installed what-so-ever.

If the AV's disappear then you can be more confident that the problem is somehow related to a shell extension. Then by installing the known shell extensions onto the test machine one-by-one until the AV's re-appear, you may isolate the culprit and decide what to do about it... if it's one that your users/customers are unlikely to be using then you might choose to simply list it as a known compatibility issue and move on to other issues.

If the AV's do not disappear however, then you can pretty much rule out the Delphi dialogs or any shell extensions as being somehow responsible at all.

More generally, it would be most helpful to see the code in which the AV is occurring, if at all possible.

Addendum:

I did find this reference to AV's occuring with the Common Dialog components themselves. This though would not count as "a strange place" for an AV, being consistently reproducible - it seems - within the dialog components themselves. But I thought I would mention it anyway. Without knowing precisely where your AV's are occurring it is possible that this might be related.