How can I make my Selenium tests less brittle?

Question

We use Selenium to test the UI layer of our ASP.NET application. Many of the test cases test longer flows that span several pages.

I've found that the tests are very brittle, broken not just by code changes that actually change the pages but also by innocuous refactorings such as renaming a control (since I need to pass the control's clientID to Selenium's Click method, etc) or replacing a gridview with a repeater. As a result I find myself "wasting" time updating string values in my test cases in order to fix broken tests.

Is there a way to write more maintainable Selenium tests? Or a better web UI testing tool?

Edited to add: Generally the first draft is created by recording a test in the IDE. (This first step may be performed by QA staff.) Then I refactor the generated C# code (extract constants, extract methods for repeated code, maybe repeat the test case with different data, etc). But the general flow of code for each test case remains reasonably close to the originally generated code.

Answer 1

How are you creating your Selenium tests, by recording them and playing them back? What we have done is build an object model around pages so that you call a method like "clickSubmit()" rather than clicking on an id (with a naming convention for these ids), which allows selenium tests to survive many changes.

Answer 2

There are no innocuous changes when it comes to test automation ;)

We use the SAFS framework with Rational Robot (RRAFS) to minimize impact to our automation scripts. There's still work to maintain the application map, but the scripts remain stable for the most part. The SAFS framework sounds very similar to the method cynicalman mentions, but already packages up the generic methods you would use in your scripts.

The SAFS site says there's partial support for Selenium, so this may work for you.

Answer 3

I've found that using XPath expressions in Selenuium-RC adds alot to the robustness of a test.

I write my tests in a similar manner. The first pass is often written via the IDE/Record to get most of my page-flow and click operations. Once I've got that, I begin stepping through the test via Selenium-RC adding assertions and changing absolute widget locators to more readable and friendly Xpath expressions. (as well as documenting the test! :) )

One thing to be aware of.. if your tests are xpath-heavy, they may run a little slower in IE6 due to its poor javascript execution abilities. (I have some test suites that take almost an hour longer to execute under IE than under FF. It's managable, but just something to keep in mind when you're writing the tests.)

Answer 4

You may or may not be able to write tests that are resilient to refactoring. Here's how to make the refactoring less painful: Continuous integration is essential.

Run them every day or every build. The sooner it's fixed, the easier.

Ensure devs can run the tests themselves. Again, the sooner it's seen and fixed, the easier.

Keep selenium tests few. They should focus on critical path / pri 1 test scenarios. Deep testing should be done at unit test level (or jsunit tests). Integration tests are always expensive and less valuable.

Answer 5

Hello Helen,

I've found PageObject pattern very helpful.

http://code.google.com/p/webdriver/wiki/PageObjects

more info: - http://stackoverflow.com/questions/616810/whats-the-point-of-selenium/616887 - http://stackoverflow.com/questions/99876/selenium-critique/211452

maybe a good way to start is to incrementally refactor your test cases.

I use the same scenario you have selenium + c#

Here is how my code looks like:

A test method will look like somethink like this

    [TestMethod]
    public void RegisterSpecialist(UserInfo usrInfo, CompanyInfo companyInfo)
    {
        var RegistrationPage = new PublicRegistrationPage(selenium)
              .FillUserInfo(usrInfo)
              .ContinueSecondStep();
        RegistrationPage.FillCompanyInfo(companyInfo).ContinueLastStep();
        RegistrationPage.FillSecurityInformation(usrInfo).ContinueFinishLastStep();
        Assert.IsTrue(RegistrationPage.VerifySpecialistRegistrationMessagePayPal());
        selenium.WaitForPageToLoad(Resources.GlobalResources.TimeOut);
        paypal.LoginSandboxPage(usrInfo.sandboxaccount, usrInfo.sandboxpwd);
        Assert.IsTrue(paypal.VerifyAmount(usrInfo));
        paypal.SubmitPayment();
        RegistrationPage.GetSpecialistInformation(usrInfo);
        var bphome = new BPHomePage(selenium, string.Format(Resources.GlobalResources.LoginBPHomePage, usrInfo.AccountName, usrInfo.Password));
        Assert.IsTrue(bphome.VerifyPageWasLoaded(usrInfo));
        Assert.IsTrue(bphome.VerifySpecialistProfile());
        bphome.Logout();
    }

A page Object will be something like this

public class PublicRegistrationPage
{
    public ISelenium selenium { get; set; }

    #region Constructors
    public PublicRegistrationPage(ISelenium sel)
    {
        selenium = sel;
        selenium.Open(Resources.GlobalResources.PublicRegisterURL);
    }
    #endregion
    #region Methods

    public PublicRegistrationPage FillUserInfo(UserInfo usr)
    {
        selenium.Type("ctl00_cphComponent_ctlContent_wizRegister_tUserFirstName", usr.FirstName);
        selenium.Type("ctl00_cphComponent_ctlContent_wizRegister_tUserLastName", usr.LastName);
        selenium.Select("ctl00_cphComponent_ctlContent_wizRegister_ddlUserCountry", string.Format("label={0}",usr.Country ));
        selenium.WaitForPageToLoad(Resources.GlobalResources.TimeOut);
        selenium.Type("ctl00_cphComponent_ctlContent_wizRegister_tUserEmail", usr.Email );
        selenium.Type("ctl00_cphComponent_ctlContent_wizRegister_tUserDirectTel", usr.DirectTel);
        selenium.Type("ctl00_cphComponent_ctlContent_wizRegister_tUserMobile", usr.Mobile);
        return this;
    }

}

Hope this helps.