Why is Selenium RC so slow?

Question

Hi.

For some time I have been investigating Selenium RC in order to do functional testing of my web application. I have now found a test strategy that is so effective, that I do not want to move away from Selenium RC (after spending weeks trying to figure out a good way to validate ASP.NET validation controls).

But now that my Selenium RC adventure is moving from a POC to be something that I actually use, I'm running into a problem. It is insanely slow. Executing a single test that loads a page, fills in some fields, and clicks a button takes in the magnitude of seconds to execute. When it is executing, I can easily see each individual field being filled out one at a time. Using Selenium IDE in Firefox is not that slow.

I found this page, that clearly specifies that Selenium RC is slow http://selenium-grid.seleniumhq.org/how_it_works.html

But why is that? Is it because the browser is polling the selenium server? If so, can this polling interval not be modified? Or is there another reason. I am not accustomed to a remote call taking a humanly noticable amount of time to execute.

It is horrible that executing a few tests should take so long. I can execute my entire presentation (MVP), business, and database layer test suite (500+ tests) way quicker than it takes to run 10 tests for a single web page.

Answer 1

Functional/Integration tests will take longer to run especially since they are running in a Browser. This means that it they have to load up all 3 layers of your MVC and then execute and the same when it is doing anything on the page. So every action has the potential to go down to the database. This is inherently a long running tasks compared to unit tests.

The tests start by doing an open on that page which then waits for everything to load. So if this is taking a long time then it could be taking a long time for your user if they were to access the page. E.g. Lots of images, unminified JavaScript/CSS, poor expiries on downloads.

What that page from Selenium is saying that the server is a bottleneck because it implies that you are running the tests synchronisely and if you moved to Selenium Grid it can run them in parallel to make the test suite complete faster. It is not suggesting that the selenium server is polling to see what it should do but instead the Selenium Servers poll the Grid hub to see if it is still alive and to show they are still alive.

The other reason the tests are running slow is Selenium's base language is JavaScript which interacts with the DOM. The DOM can slow things down a lot especially if your tests are using XPath as locators. XPath + JavaScript + IE + Selenium == Painful and there is nothing that we Selenium Developers can do more to fine tune it. Well there is and that is going to be Selenium 2 which is in alpha and can be downloaded from http://selenium.googlecode.com/ . I have been working on the .NET implementation am seeing huge speed improvements at the moment. I have blogged about it because the changes astounded me. I was seeing upto 8 tests running in the same time it used to take Selenium 1 to run 1 test

Answer 2

What operating system are you using? We're having similar issues.

On Windows 2003 Server, x86. Everything ran nice and quickly.

However, since the switch to Windows 2008 Server, x64 our automation tests now take a long time.

We've figured out the rough area where the problem lies. If we set selenium server running, we can fire Http Requests at it. So we built a quick app to send the request, and time how long it takes to get a response.

If we send invalid requests, we get a response in around 9ms. If we send a valid request, it takes 500ms to respond.

So from the look of things, it's something to do with Selenium Server running on 2008, or a 64-bit version of Windows. The problem looks like in lies with Selenium Server though (or possibly the JRE).

I'll do an update if we figure it out.

PS We're using the C# selenium lib, but in the test app above, we're hitting it direct (have tried using other languages)

Answer 3

Hi, we have been experiencing the same problems. It seems to be something to do with how the selenium server keeps connections open. We found if we pinged the server with an invalid URL it would return a 500 however if we pinged it with a valid url it would hold onto the connection for ~500ms. We could replicate the issue by pinging another site but found setting HttpWebRequest.KeepAlive = false would fix it there. Annoyingly when we applied the same fix to the Selenium RC code it didn't make a difference. To make matters worse, if you run the selenium server on windows server 2008 64bit and the client on windows server 2003 32bit it runs quickly. So the issue seems to be with the server but its the client which is sensitive to it... bit of a head scratcher.

My guess is that there is an issue with the Windows server 2008 networking stack (It was rewritten for Server 2k8/Vista) so for the time being we will be running our tests on Server 2k3. In the long term we are looking at migrating to Selenium 2.0 which merge's in Google's WebDrivers which talk directly to the browsers API's rather than through JS which will get rid of the slow issue since no http requests are being made. Selenium 2.0 is unfortunately still in alpha so might have to wait a bit...

Answer 4

Is it possible that your default execution speed is too low? Check out getSpeed() and setSpeed() methods on DefaultSelenium.

Answer 5

I have tried Selenium 2 standalone server to run my scripts and testing speed was just as slow as 1.0.3 server. This a real problem and there are ways to mitigate it to some extent with grid, but it shouldn't be that way...

Answer 6

I tried no improvements in the speed at all.Its blazing through the pages in ide, but as soon as I go through the server its sitting and loading waiting for pages to finish loading for minutes and my steps are timingout. I'm probably better off writing some sort of script to automate process of opening test suites and running them from IDE than using the server as its really a deal killer.

It seems like I'm not the only one having this issue. Did I misread somewhere or S 2 was using completely different logic/framework to execute the test?

Answer 7

I have the same issue on windows 7 64bit.

The strange thing is, I actually have one Windows 7 64bit system on which tests are running fast, and one windows 7 64bit version which has the same issues discussed in this question.

The system on which tests are running slow is actually alot faster then the system on which the tests are running superfast.

On both systems i run the same version of firefox, the same version of java and the same testcode. I disabled NOD32 on the slow machine to see if it would help but it didn't. So i'm still in the process of figuring out what's different between these 2 machines. If i find anything i'll post it here.

edit

In the end I've worked around this issue by using a windows xp mode virtual machine. Both my unittests and selenium RC are run on the virtual machine and now everything is superfast again.

edit 2

I've found out that I didn't disable the right thing in my NOD virusscanner, after disabling a thing called 'web access protection' the problem was solved!

Answer 8

We are after the same problem in various environments. I'll definitely come back and post my results as soon I'll find something. Please do the same if you'll find a potential solution.

Answer 9

I'd agree that there is definitely something wrong with the communications between client and server - 3 second long functional tests run using the Selenium IDE are taking on the order of 30 minutes (if I set timeouts long enough that everything doesn't time out) when run on on two identical machines with one acting as server and the other running RC standalone.

I could understand a 3 second test case maybe taking 12 seconds under RC, just for some network delays, but 900+ seconds is ridiculous!

Answer 10

Are you testing with IE and Selenium in multiwindow mode? This is extremely slow and you should try to start the seleniumserver with -singlewindow

Answer 11

Hi.

Mind if i join?

We have the same problem. And nobody said that the guy uses Firefox and it is slow. The header states that he uses IE and it is slow.

so stick to that when posting please.

We have discovered that xpath is a problem with IE because it does not have a native library for it?!

But we are not sure so... We have to investigate the problem further.

Any luck anyone else?