Can I use Watir to scrape data from a website on a linux server without monitor?

Question

Can I use Watir to scrape data from a website (AJAX used) but on a linux server without monitor? (linode.com) ?

Answer 1

There are several ways to do this:

1. Use HtmlUnit, either Celerity or watir-webdriver (through the remote Selenium2/WebDriver server).

2. Use a real browser + a virtual X server (Xvfb). I'd recommend using watir-webdriver's Firefox driver and the Headless gem for a simple way to control this from Ruby.

This is basically a tradeoff between speed and realism. Personally I'd go with #2 if the site has any complex JavaScript or invalid HTML, but both approaches could be worth investigation.

For the future, I'm keeping an eye on this project, which looks like a terrific idea.