Selenium Critique

Question

Hi all,

I just wanted some opinions from people that have run Selenium (http://selenium.openqa.org) I have had a lot of experience with WaTiN and even wrote a recording suite for it. I had it producing some well structured code but being only maintained by me it seems my company all but abandoned it. If you have run selenium have you had a lot of success? I will being using .NET 3.5, does Selenium work well with it? Is the code produced clean or simply a list of all the interaction? (http://blogs.conchango.com/richardgriffin/archive/2006/11/14/Testing-Design-Pattern-for-using-WATiR_2F00_N.aspx) How well does the distributed testing suite fair?

Any other gripes or compliments on the system would be greatly appreciated!

Answer 1

If you are using Selenium IDE to generates code, then you just get a list of every action that selenium will executes. To me, Selenium IDE is a good way to start or do a fast "try and see" test. But, when you think about maintainability and more readable code, you must write your own code.

A good way to achieve good selenium code is use the Page Object Pattern in a way that the code represents your navigation flow. Here is a good example that I see in Coding Dojo Floripa (from Brazil):

public class GoogleTest {

    private Selenium selenium;

    @Before
    public void setUp() throws Exception {
            selenium = new DefaultSelenium("localhost", 4444, "*firefox",
                            "http://www.google.com/webhp?hl=en");
            selenium.start();
    }

    @Test
    public void codingDojoShouldBeInFirstPageOfResults() {
            GoogleHomePage home = new GoogleHomePage(selenium);
            GoogleSearchResults searchResults = home.searchFor("coding dojo");
            String firstEntry = searchResults.getResult(0);
            assertEquals("Coding Dojo Wiki: FrontPage", firstEntry);
    }

    @After
    public void tearDown() throws Exception {
            selenium.stop();
    }

}


public class GoogleHomePage {

    private final Selenium selenium;

    public GoogleHomePage(Selenium selenium) {
            this.selenium = selenium;
            this.selenium.open("http://www.google.com/webhp?hl=en");
            if (!"Google".equals(selenium.getTitle())) {
                    throw new IllegalStateException("Not the Google Home Page");
            }
    }

    public GoogleSearchResults searchFor(String string) {
            selenium.type("q", string);
            selenium.click("btnG");
            selenium.waitForPageToLoad("5000");
            return new GoogleSearchResults(string, selenium);
    }
}

public class GoogleSearchResults {

    private final Selenium selenium;

    public GoogleSearchResults(String string, Selenium selenium) {
            this.selenium = selenium;
            if (!(string + " - Google Search").equals(selenium.getTitle())) {
                    throw new IllegalStateException(
                                    "This is not the Google Results Page");
            }
    }

    public String getResult(int i) {
            String nameXPath = "xpath=id('res')/div[1]/div[" + (i + 1) + "]/h2/a";
            return selenium.getText(nameXPath);
    }
}

Hope that Helps

Answer 2

I'm using Selenium Remote Control in order to test ASP.Net apps (which is what I'm assuming you'll be targetting as well), and it works great.

If you've never used Selenium, watch some of the screencastsscreencasts for using Selenium IDE. This will give you a good idea of how 'Selenium' works. The IDE is a firefox plugins that basically lets you develop quick record-and-play tests as you go. For larger test suites though, or for writing really maintainable tests though, I'd recommend Selenium Remote Control. (The IDE is terrific if you're just getting a start though.)

Selenium Remote Control lets you use your favourite language and unit testing framework to drive a web browser in order to execute your tests. If you're most comfortable with C#/NUnit, you can write your tests that way and use all the NUnit goodies that you like. (For example, the Test-Driven.net plugin). Also, since your tests are written in a high level language, you're able to do things like inherit from a particular test class which you can use to make your actual test method code much cleaner. (Or at least thats the way I write my tests. It lets me test complex scenarios which keeping my test method line-count at a reasonable number.)

You mention distributed testing. Unfortunately I haven't found a way to use the Selenium Grid project with NUnit. Selenium Grid allows you to execute your test suite over a number different machines and browser instances. So rather than running through say 200 test methods one after another (ie, serially), you could spread the load out over say four Grid instances (ie, running in four different browsers instances at a time) on a single machine or multiple machines depending on how distributed you want to get.

If you write your tests in Java or PHP though, you might have better luck. I'm expecting this to be available via NUnit with the release of NUnit2.5 which will include pNUnit for parallel testing.

If you have any further questions about selenium, just clarify your original question and I'll be happy to try and help you out. (Selenium is just one of those tools that I use everyday so I enjoy helping to get new people started with it..)

Answer 3

I started out with Selenium IDE and Selenium Core. Those are definitely good tools to get you started. But they're not very powerful, since you can only use Selenese, Selenium's HTML-based command-by-command language.

Now I use Selenium Remote Control with the Ruby driver, which allows me to utilize what Ruby offers. I test many environments: Windows 2000, XP, Vista, Mac 10.4/10.5, and for each of those that apply, Safari 2/3, Firefox 2/3, Internet Explorer 6/7.

Selenium claims to be compatible with all those OS's and browsers, though I'm having problems currently with Internet Explorer (my first question on StackOverflow is about that, actually). But I don't know of any other tools that are this powerful and works with so many platforms.

The biggest problem I've had with Selenium is the DOM parsing. JavaScript's childNodes is unreliable because Safari/Firefox ignore whitespace & comment nodes, while Internet Explorer doesn't. XPath in Internet Explorer is 10-20 times slower than in SF/FF. innerHTML isn't always reliable in IE.

Answer 4

Selenium is pretty decent tool but there are couple things to watch out:

• Selenium IDE and Selenium core do not share 100% same functionality. For example right clicking is supported by IDE but current core release does not have it. However, using a newer version from their repository solves that.

• In case of ext js, gwt etc make sure you have proper IDs for your display elements instead of automatically generated (random) ones.

• Maintaining test cases. I have seen cases where a lot of effort was put on Selenium tests and good coverage. Later on tests started to fail as the person creating them was busy with other tasks and no-one else wanted to touch them. But this was issue with management, not Selenium.

Answer 5

I'm a big fan of Selenium. One major gotcha that's good to know about ahead of time, though, is that Selenium IDE has a lot of trouble with pop-up windows. These problems don't persist in Selenium RC, but it can make development a bit of a headache.