saving / mirroring / crawling web pages that use javascript to generate content

Question

I want to download web pages that use javascript to output the data. Wget can do everything else, but run javascript.

Even something like:firefox -remote "saveURL(www.mozilla.org, myfile.html)"

would be great (unfortunately that kind of command does not exist).

Answer 1

I'd look at the selenium browser automation tool (http://seleniumhq.org/) - you can automate visiting a webpage, and saving the resultant HTML.

We used it to great success for a similar purpose on a prior project.

Answer 2

If it can be a Windows based app, you can try using the browser component of any programming language like C#, Visual Basic, Delphi, etc to load the page and then peek into the content and save it. The browser component should be based on IE rendering engines and should support JavaScript. There's a question regarding snapshots of websites here. May be of some use to you.

Alternately, you could consider building your own Firefox extension. Take a peek here for further details (there's no "next" button, just the menu on the left for navigation, confused me at first).

Answer 3

I second Alex's suggestion for Selenium. It runs in the browser so it can capture output HTML after Javascript has modified the DOM.

Answer 4

The problem with using a browser-driven approach is that it'll be hard to automate the process of scraping.

Look for a "headless browser" in your favourite programming language of choice. Alternatively, you can use Jaxer to load the DOM serverside, execute the JavaScript and let it manipulate the DOM, and then scrape the modified DOM using the same JavaScript you are already familiar with. This would be my preferred approach.

Answer 5

I have done this before using:

• chickenfoot
• webkit