Why can't I register my COM DLLs?

Question

My application needs two COM DLLs to be registered. It is done automatically if the user has the necessary access rights, but otherwise it can be done using regsvr32.

Now on a few workstations the following happens:

1. Start cmd.exe as Administrator
2. Register first DLL: Success
3. Register second DLL: Failure (0x80004005, Access denied)

The first time this error was reported to me, I automatically responded: "Of course it doesn't work, you need Administrator privileges." Needless to say, I felt pretty embarrassed as I tried to register the DLL myself...

What could be the reason for this: One DLL can be registered, the other not.

Some background information:

• Both DLLs are created with Delphi 2005.
• Both DLLs have very simple interfaces and only very few classes.
• The failing machines were always Vista or Windows 7.

Normally I'd start Process Monitor, but I don't have access to one of the workstations right now, so I have to gather as much information as possible until I can get my hands on one of them.

UPDATE: I remotely accessed one of the failing workstations. Mysterious things happened:

• Starting Process Monitor was not possible (!) After a few seconds Windows showed a "not enough memory" error.
• Switching to a different user made it possible to register the DLL. (!)

In other words:

• Log on as UserA, start cmd.exe as Administrator, regsvr32.exe: Failure
• Log on as UserB, start cmd.exe as Administrator, regsvr32.exe: Success

What could introduce such behaviour?!

Answer 1

Couple of things to try and check for:

1. Run process explorer on a machine where both dll will succeed registering and do quick comparison of the reg keys and file locations both DLLs touch. There might be some difference that would explain this behavior
2. Is it always the same dll that fails? What happens if you reverse the order of the registrations? It's possible that the first one is affecting the second one (by setiing an ACL on a reg key for example)
3. It might be interesting to collect some statistics on whether this is 32-bit vs. 64-bit issue. There are certain specifics on HKCU and HKLM behavior on 64-bit Vista and Win7, whcih might affect the registration.
4. Ask users that hit this problem, if they attempted registration as regular user before doing it as an admin. The HKLM virtualization could be playing bad tricks if this is the case.

Update

Based on your comment, it looks that failed registration attempt as a regular user is leaving the machine in a messed up state. It's something either in HKCU\Software\Classes (the per-user mapping of the HKCR) or in HKCR\VirtualStore\MACHINE\SOFTWARE (the virtualized HKLM). You need to determine exactly what is being left in the registry upon unsuccessful registration attempt.

I would run process explorer on a clean machine and attempt non-elevated registration to take a snapshot.

Also, try getting the machine in such a state and then run non-elevated unregister. In the ideal case this should "fix" the machine. If that works, you have a quick workaround you can give to your users while you are working on the proper fix.