What are good test automation frameworks?

Question

I am looking for a tool to integrate many different types of automation and test frameworks in order to automate fully end-to-end scenarios for testing. For example, a typical workflow might be:

• Install an Operating System (Ghost, etc..)
• Configure the operating system (open firewall exceptions, setup users, etc..)
• Install Software (Visual Studio, SQL, etc..)
• Run tests (Visual Studio Unit Tests, nUnit, custom tools, etc..)
• Generate Reports on the above runs
• Allow monitoring of machines, tests in progress, and scheduling of test runs

Does anybody know what system like this might exist?

Answer 1

I would question the scope and of what you are trying to achieve here, and perhaps suggest you need to break it down into a set of well-defined test strategies with clear goals.

For example, I cannot see any reason why you would want to install an entire OS, development environment, etc every time you run unit tests. This should be something individual developers do in short iteration cycles in their own development environments.

Whereas something like deployment testing (where it could make sense to test a full user installation of your built application and host OS from scratch) would be attempted less frequently and require a completely different set of tools to those you would need to perform unit testing.

Really it's all about separating out what your trying to achieve and how you are attempting to test at any given stage of the development life-cycle and build specific standalone test processes around those scenarios. This is probably preferable to over-architecting something which does "everything" in a single work flow.

Answer 2

I think for the first three steps you are looking for a way to start with a well-defined system configuration for running your tests. A virtual machine (VM) manager like VMware or Xen is the way to go for this; you can store multiple VM snapshots with the exact configurations you need for different deployments. You can also automate the launching of the VMs via an API which you can hook up to your build system.