Selenium vs. SimpleTest vs. WatiN

Question

It seems there is very few comparison between Selenium / WatiN and SimpleTest (which has web testing features too).

I tried Selenium and found the GUI great to create tests as you can see what's going on and record without typing all commands manually.

As for running the tests, Selenium is way more complex than SimpleTest. For SimpleTest you just have to run a PHP script which does all the tests (client or browser side). This means that whatever browser or browser settings you're using, you can simply go to the test URL and it'll work just the same.

What would be useful here are some comments from people that used Selenium or WatiN: Why is Selenium so famous? In other words, what would be the main benefit of using Selenium for example, over SimpleTest?

PS: Please exclude reasons like "SimpleTest requires PHP"; that's pretty obvious is PHP is included in most LAMP anyway.

Answer 1

There is a major difference between SimpleTest's web-tester and the Selenium suite :

• Selenium works with a real browser :
  • Each time a test is launched, a real Firefox, or a real Internet Explorer, is launched
  • All the browsing is done in that real browser
  • Which means you get all the Javascript features from the browser -- i.e. you can test fully-dynamic webpages.
• SimpleTest's web-tester simulates a browser :
  • An HTTP request is sent, the HTML content is fetched
  • And that HTML content is parsed
  • There is some level of features to test cookies and forms ; but nothing about Javascript

A couple of consequences and thoughs :

• Selenium tests take a lot of time to execute : launching a browser, and surfing (which includes downloading the JS/CSS/images, display everything including the ads, ...), is long and slow
• SimpleTest tests should be faster : only send the HTTP request, parse the HTML, and that's it
• Selenium requires a machine with a graphical interface, to launch the browser, which is a graphical software ; which also means it's harder to install/configure
• Selenium allows you to test rich/dynamic/javascript-based applications much better that only fetching and parsing HTML
• If you are testing a PHP application, PHPUnit is generally considered much better (it's activelly maintained and developped, at least) that SimpleTest ; and PHPUnit has support for Selenium tests

I would recommend a combinaison of both, if you can :

• Test HTML-related stuff with SimpleTest (Or Zend_Test, if using Zend Framework) :
  • it'll be faster
  • those tests will not depend on a browser
• Test dynamic pages with Selenium

i.e. use the best of both tools ;-)

Answer 2

Since SimpleTest only deals with the HTML content of a page you can't test pages with it that rely on JavaScript behavior. At the end of the day it's a speed and functionality tradeoff.

• If you need JavaScript functionality or want to test browser-specific behavior, use Selenium.
• If you want speed and have static HTML pages, use SimpleTest.

BTW, Selenium can be integrated into a PHPUnit test suite: http://www.phpunit.de/manual/3.1/en/selenium.html

Answer 3

Just to add another option, TestPlan works with both the Selenium back-end and HTMLUnit, so it can be used with our without a browser. The scripting language is simple and allows fast creation of automation tasks.

The browserless backend supports JavaScript very well, but for those cases where it just doesn't work you just switch to the Selenium mode and use a real browser.

Answer 4

Also note that simpletest needs the drupal codebase to be patched. You cannot test your site on an exact mirror of your production site.

Regards.