How can I use Perl to scrape a website that reveals its content with JavaScript?

Question

I need to write a Perl script to scrape a website. The website can only be scraped with JavaScript, and the user is on Windows.

I got some way with Win32::IE::Mechanize on my work machine, which has IE6, but then I moved to my netbook which has IE8, and can't even get as far as fetching a simple page.

Is Win32::IE::Mechanize up to date with the latest versions of IE?

But, more to the point, given a recent WinXP machine, what's the quickest, easiest way to scrape a site which only reveals its content via JavaScript?

Answer 1

Have a look at Win32::Watir. It's a newer module and explicitly supports IE 6, 7 and 8.

Answer 2

I don't see any mention of WWW::Mechanize, so I'll bring it up just for completeness. Selenium is also becoming very popular and can be used in a lot of testing scenarios.

Answer 3

WWW::Selenium.

• It allows you to specify which browser to use (IE and Firefox are supported from the get-go)
• It supports access to elements via xpath elements, table IDs, text (regex-matching!) and URLs
• It provides a Swiss army knife of user-interaction options, giving you flexibility over how you wish to simulate end-user browsing

You'll need to download the Selenium Remote Control and have it running in the background for the module to work.

It may not be a good option if your page load times are unpredictable.

Answer 4

WWW::Scripter and its ::Plugin::Javascript can probably help you.