We've got some code that uses LoadLibrary and GetProcAddress to implement a plugin architecture for one of our products. We ensure that the DLL about to be loaded is signed with our code-signing key.
We're changing the plugin architecture to use COM instead. Is there a way to enforce code-signing (preferably with our certificate) when instantiating a COM object?
I don't think you can do this directly, but you could look up the DLL that the CLSID uses in the registry and check its signature before issuing the CoCreateInstance call.
You need to do this at the DLL level using the Authenticode API. The standard API is called WinVerifyTrust() and there are samples documented there. There's another KB article number 323809 that gives an example of how to peel other details out of the authenticode information attached to your DLL.
Of course, these APIs expect to be handed a path to the DLL itself whereas in a COM plugin scenario you usually don't directly touch that but instead rely on registration to find the right binary. You can either hand-roll your load scenario (i.e. load the DLL using LoadLibrary() and call DllGetClassObject() yourself) or simply require users of your API to adhere to additional rules such as placing the DLL in a certain location regardless of registration.
Or as Rob Walker suggested, look up the CLSID registration in the registry yourself and use that as the way to find the right DLL to verify.