How to automate installer testing

Question

I'm wondering if anyone has any best practices for automating the testing of installers on various machines with potentially different hardware / software profiles and by specifying various options to the installer. The idea would be that I could write "unit test like" code to set up a machine, run the installer, then test that certain things are true. Tests might look similar to:

Test:
    Boot Machine without IIS
    Run Installer
    Assert Installer Had Errors

Test:
    Boot Machine with IIS
    Run Installer
    Assert Installer Ran

Test_Fixture:
    SetUp:
        Boot Machine with IIS

    Test:
        Run Installer without IIS install
        Assert Website Not Installed

    Test:
        Run Installer with IIS install
        Assert Website Installed

I know I could create lots of VMs, but waiting for a VM to boot for each functional test sounds like way more work than I want. What I really want is a way to virtualize the installer environment. Any suggestions?

Answer 1

If you have the installer runnable from the command line, it's easy to have a script to call it automatically.

Then you can use a web app testing tool to see it the install was successful, like this one http://seleniumhq.org/ For this you will need an unique way to test a new install - like a page with the current version.

Answer 2

We have created a set of VMs and find it is very easy to manage. We run the tests for 13 different Windows installers over night. The VMs we have created our very bare bones, so it is possible to run a number of tests in parallel.