How can I create an automated test for valid WIndows installer certificates?

Question

We have a situation where for some reason the certificates on our Windows Installers for our product tends to get broken or go bad. Never mind why, the key is that it is not currently detected by our nightly test runs.

So how can that be done?

I.e., an automatic test that runs a Windows installer and checks if it pops up a UAC or bad cert warning in some other way (depends on the setup of Windows exactly how a bad cert is reported, in my experience). Something that can be run as part of a scripted large batch of tests, and report success or failure without a human involved.

I tried searching on stack overflow, but I could not find any other question dealing with this particular issue.

Answer 1

A couple stabs at the idea:

• Use a automated test mechanism that runs GUI testing. SilkTest & WinRunner are expensive commercial options but there are some free open source things that would do the trick. The tool should be able to act like a user, run the install, and then yell foul when warnings appear. Then I think you can run a test thread to uninstall the install to return to start state for the next night.
• Verify the certificate independantly, without actually doing the install. If you know that your errors are coming from incorrect signatures, expired certificates, certificate configuration, or something that is part of the certificate and/or how the certificate secures the install - you should be able to run that test as a program using a common API - for example, Java's PKI stuff should be able to handle this.

The first option will test exactly the actions you are trying to do, but it involves introducing automated GUI tests into your test batch, which may not be ideal for you. And sometimes automated test tools for this get funny between OS versions or other things that change GUI.

The second option will test the qualities of the certificate and signature, but they won't cover whether Windows will really let you install it. There are some other types of errors it may not catch - for example, if your host requires OCSP to verify the certificate and your OCSP is down, the windows install would fail, but your certificate test would pass (unless you build an OCSP check into it...).