I'm going to start building some automated tests of our presentation soon. It seems that everyone recommends WatiN and Selenium. Which do you prefer for automated testing of ASP.NET web forms? Why did that product work better for you?
As a side note, I noticed that WatiN 2.0 has been in CTP since March 2008, is that something to be concerned about?
I've used both, they both seem to work ok. My nod is for Selenium as it seemed to have better Ajax support. I believe WaTiN has matured though since last I used it so it should have the same thing.
The biggest thing would be which development environment do you like to be in? Selenium and Watin have recorders but Selenium is in the browser and watin is in visual studio. + and -'s to both of those.
I generally use Selenium, mainly because I like the Selenium IDE plugin for FireFox for recording starting points for my tests.
You will have to do both if you need to do IE and FF testing, but they are only going to work so well for presentation testing. They cant detect if one element is slightly off, just that the elements are present. I dont know of anything that can replace the human eye for UI / presentation testing, though you could do a few things to assist it (take screenshots of the pages at each step for users to review).
I use Watin, but haven't used Selenium. I can say I got up and running quickly on Watin and have had few to no problems. I can't think of anything I have wanted to do that I couldn't figure out with it. HTH
I recommend WebAii since that's what I've had any success with and when using it my gripes were few. I never tried Selenium and I don't remember using WaTiN much, at least not to the point where I could get it to succesfully work. I don't know of any framework that deals with Windows dialogs gracefully, although WebAii has an interface for implementing your own dialog handlers.
Hi,
Just want to say that I'm currently working hard on a beta release of WatiN 2.0 somewhere in Q1 of 2009. It will be a major upgrade to the current CTP 2.0 versions and will basically give you the same functionality to automate FireFox and IE as version 1.3.0 offers for automating IE.
So no concerns there.
Hope this helps in making your choice Jeroen van Menen Lead dev WatiN
We've tested both and decided to go with WaTiN. As others have pointed out, Selenium does have some nice features not found in WaTiN, but we ran into issues getting Selenium working and once we did it was definitely slower when running tests than WaTiN. If I remember correctly, the setup issues we ran into stemmed from the fact that Selenium had a separate app to control the actual browser where WaTiN did everything in process.
The biggest difference is that Selenium has support for different browsers (not just IE or FF, see http://seleniumhq.org/about/platforms.html#browsers.
Also, Selenium has a remote control server (http://seleniumhq.org/projects/remote-control/), which means that you don't need to run the browser on the same machine the test code is running. You can therefore test your Web app. on different OS platforms.
In general I would recommend using Selenium. I've used WatiN few years ago, but I wasn't satisfied with its stability (it has probably improved by now). The biggest plus for Selenium for me is the fact that you can test the Web app. on different browsers.
If you're looking to make a serious long-term investment in a framework that will continue to be improved and supported by the community, Selenium is probably your best bet. For example, I just came across this info on Matt Raible's blog:
As of Friday, Google has over 50 teams running over 51K tests per day on internal Selenium Farm. 96% of these tests are handled by Selenium RC and the Farm machines correctly. The other 4% are partly due to RC bugs, partly to test errors, but isolating the cause can be difficult. Selenium has been adopted as the primary technology for functional testing of web applications within Google. That's the good news.
I also went to one of the Selenium meetups recently and learned that Google is putting serious resources into improving Selenium and integrating it with WebDriver, which is an automated testing tool developed by Simon Stewart. One of the major advantages of WebDriver is that it controls the browser itself rather than running inside the browser as a Javascript application, which means that major stumbling blocks like the "same origin" problem will no longer be an issue.
At the risk of going off on a tangent, I'd recommend Axe/WatiN. Axe allows tests to be written in Excel by 'Manual' Testers with no knowledge of the underlying test 'language'. It does need a 'Technician' to write the bespoke actions (IE. Today I had to do a slightly complex Table lookup & cross-reference) but once written the actions can be used in tests by the non-techy testers.
I also heard that the UK Government Gateway project (which I believe has 6K+ tests automated tests) recently ported all their tests from Axe/Winrunner to Axe/Watin within a week!! And many of the tests are pretty complex - I know as I worked on it a few years ago...
I'm looking at Selenium at the moment, as a potential Client uses it. But i do suggest a wee look at Axe as a layer above the 'work horse' tool.
If you have to access iframes, modal dialogs and cross domain iframes WatiN is a way to go. Selenium couldn't handle the iframes it was throwing commandtimeout exceptions. WatiN you could do lot more things especially if the website uses IE specific stuff like ShowModalDialog etc.. WatiN handles all of them very well. I could even do cross domain iframe access.