Problem in hosting ActiveX on Vista (in a Visual Studio 6 C++ application) II

Question

Hi, (I'm re-posting this message since I signed up as a user now)

I just installed Visual Studio 6 (SP5) on a Vista Enterprise machine. Had some problems but I think it’s set up alright.

The problem is my VC++ 6 application fails when trying to host an ActiveX. I was able to compile it ok, though got a warning message from Vista about the rc.exe (“This program has known compatibility issues” ).

When I debugged it I saw that my class that derives from CAxDialogImpl fails on its Create() method. The same application worked just fine on XP!

Is there a known compatibility issue there?

Some tech info: I saw that CAxDialogImpl::Create() (I pass NULL here) calls AtlAxCreateDialogA, which in turn calls CreateDialogIndirectParamA which throws a general exception.

Thanks a lot, Erik

PS - 1. I am Admin on my machine. OS is 32 bit. 2. I know that VS6 is not supported on Vista but I had no choice since I ran into troubles trying to migrate it to VS 2008.

Answer 1

OK. Found the reason and the solution.

It appears to be a DEP compatibility issue. It was hard to locate since when the application failed no exception / message shown.

The problem was caused due to old ATL components used in the my dll. Old versions of ATL generate machine code at runtime and then attempt to execute this code from pages of memory that were not marked as executable, thus causing a DEP violation if hardware-enforced DEP is enabled.

The difference between my XP machine and the vista machine was actually in the DEP operation - the DEP on the vista is hardware-based, which caused the problem.

Also, I had COM applications using the same dll and working fine, however .Net applications were failing because they were created with the DEP compatibility flag.

The solution was to remove the DEP compatibility flag from the .Net applications.

Cheers, Erik

PS - two links were very helpful: http://support.microsoft.com/kb/948468 http://stackoverflow.com/questions/350977/how-to-make-my-program-dep-compatible