I have a COM inproc DLL that we are using in our product. Now if someone finds out which interface and APIs we have exposed from the DLL then those APIs can be called easily.
Is there a way to stop unknown applications from calling my APIs?
Can we add some signature in COM?
Nothing prevents you from adding a "key" parameter to the methods which will just return if the key is wrong.
Very simple but will do for starters.
Other than some sort of 'key' param, you can't prevent the curious from discovering your function and then calling it. All it takes is a debugger and some patience. To be totally secure you'd have to require some sort of certificate that authorized code could obtain but all others couldn't but that would mean you're code would have to be able to verify the certificate.
The formal way of controlling use of your object is by implementing IClassFactory2 on the class factory that creates your COM objects.
Here's a link at MSDN explaining the interface.
The benefit of creating an implementation is that nobody can fetch an instance without clearing the hurdles of registration through IClassFactory2.
The downside is that you'll have to inspect all the locations where you are creating an object, to make sure that they haven't broken. Creating instances becomes more burdensome, although some languages already have facilities to make the process less painful (ex. VB6).
If you are trying to protect an object that has a lot of instantiation activity, you might want to go with Mastermind's method of adding a key parameter, or add an unlock method of some sort to your interfaces that must be called correctly before the component behind it can be used.
You can prevent your DLL from being registered by overriding DLLRegisterServer, making it impossible for someone to get the required info for object creation (unless some other app is hacked). This will make your users work harder too, but they will be able to use registration-free COM activation having the info provided by you.
You could make your interfaces inheriting directly from IUnknown (without IDispatch) and not include the type library into the DLL. This way only those who have access to the type library will be able to find what interfaces are supported and the only other way to discover the interfaces will be to just guess. If you go this way you might also wish to minimize the number of classes exposed to registry (those that can be created with CoCreateInstance()) and use a set of factory methods of some dedicated registry-exposed class instead.
This implies that only vtable early-binding will work with your component. You will also be unable to use default call marshaling with this component (since no type library is included). And this is not real protection, just a way to hide things.