How to verify a binary signed with a self-signed certificate?

Question

We want to add automatic software updates to our application, but our company isn't yet ready to buy a code-signing certificate from a trusted root CA, so we'll be using a self-signed certificate to sign code updates (.exe and .dll) for now.

Question: how to verify a binary signed with a self-signed certificate, without having to install the certificate, using Microsoft's Cryptography API? The .cer file to check against will be bundled with the application. Or is it simpler to use a generic Crypto library?

Answer 1

I seem to recall hearing of a way to enable self-signed certs some years ago, back in the Win2k days, but it was very hacky, not at all suitable for public deployment and has probably been "fixed". If you do think about using some other crypto library, or developing your own, take care: it's very hard to distinguish good crypto from bad crypto.

Answer 2

You can skip the whole X509 thing, after all you don't really need it if you're going to be using your own certificates...

For what you want to do, first you have to generate your RSA private/public key pair. Then you store the public key in your application.

When you have an update, you sign it on your site, by getting the MD5 or SHA-1 or whatever hash you want to use; then you encrypt that hash with the private key. The installed applications fetch the update and the signature (the encrypted hash); when the application gets the binary file, it computes its hash, then decrypts the other one using the public key and compares them. If they're identical then it's a valid update, otherwise you reject it and warn the user or something.

With X509 certificates that are self-signed the mechanism is going to be exactly that, but the public key is going to have a bunch of additional data like the identity of the issuer which will be the same identity of the certificate.