The ultimate browser testing machine...

Question

Hi All,

What would be the ultimate web application browser compatibility test machine be? The testing environment would be (XP: IE6,FF3.6), (XP: IE7,Chrome), (XP: IE8,Safari), (MacOsX: Safari, FF3.6, Chrome), (Ubuntu: FF3.6, Chrome)

Of course I want it all in one machine, my initial thought was Using a MacOsX as a host machine and use virtual box to install 3 xp and one ubuntu on it.

Do you think it is an optimal solution? Do you have any better solution to achieve the above?

---

Update: This testing machine will be in a network. I also need these three functionality out of my VM:

1. Need to be able to VNC via network to the testing machine.

2. All VM instances need to be able to access to the local network.

3. Somehow, I need to be able to test using third party Internet provider (To test network caching and proxies)

Answer 1

First: to get all those, it has to be a Mac (you can also install the iPhone simulator to cover mobile Safari as well).

Now, there are better VM hosts for OS X than Virtualbox... Parallels or VMWare Fusion will get you better video performance.

Get your XP as a Win 7 with downgrade license, that way you can test on Windows 7 as well.

With respect to your updates: OS X supports VNC natively, no trouble there. Similarly, any decent VM container will let the VM browse using the hosts's network connection.

Now, about using another internet provider or whatever to test network topology cases: that's a bit more complicated, but essentially you could VPN your VMs into a Linux machine or VM running a network simulation, either made up using netfilter and tc rules, or by running an ns-3 instance and creating an entirely simulated network. Or you could VPN the testing VMs into a VPN server somewhere out on the internet.

Answer 2

Consider the engines used.

Since cross-platform differences in browsers using the same engine (e.g.: Chrome, Safari) should be neglible you could get away with virtualizing 3 (IE6, IE7, IE8) and adding Safari, Chrome, FF and Opera versions to one of those virtal machines.

At one point Microsoft also distributed VM images specifically targeted at testing cross IE compatibility.

If you're aiming for ultimate coverage you should also be prepared to add Firefox 2.x, 3.0, etc...

Answer 3

I use Mac OS X and 64-bit VMWare Fusion, and I typically run Mac plus Ubuntu plus 3 flavors of Windows (XP with IE6, Vista with IE7 and Win 7 with IE 8) I used to have 9, one flavor OS per IE level, but that got to be too much maintenance, so I pared it down. I found if a browser had problems it was typically not OS-related, or at least not often enough to make me care.

Answer 4

I hope you're automating this too... run it all with Cucumber / SpecFlow and WatiN / WatiR via continuous integration like TeamCity (it's free for small time usage) or CruiseControl (less preferable)... so everytime your build runs, it delegates the acceptance tests to each virtual box.

With TeamCity, you can setup a build agent on each virtual box and have TeamCity delegate a specific build (Example: Build: Acceptance Tests Virtual Box Windows 7).