Web scraping sites that require javascript support

Question

Possible Duplicate:
Screen Scraping from a web page with a lot of Javascript

I just want to do tasks such as form entry and web scraping, but on sites that require javascript support. And I also need to enter forms, scrape, and so on in the same session. Ideally, I'd like a way to control a web browser from the command line. And I also want to only use Linux for all this, so I can't use .NET.

I found the webbrowser library for Python, but its capabilities look very limited. If that could interface with mechanize and BeautifulSoup, that'd be amazing. Any suggestions? Thanks!

Answer 1

This has been asked already.

• http://stackoverflow.com/questions/857515/screen-scraping-from-a-web-page-with-a-lot-of-javascript
• http://stackoverflow.com/questions/419457/screen-scraping-through-ajax-and-javascript
• http://stackoverflow.com/questions/532399/how-do-i-implement-a-screen-scraper-in-php
• http://stackoverflow.com/questions/125177/whats-a-good-tool-to-screen-scrape-with-javascript-support
• http://stackoverflow.com/questions/727283/are-there-command-line-or-library-tools-for-rendering-webpages-that-use-javascrip
• http://stackoverflow.com/questions/1106268/command-line-url-fetch-with-javascript-capabliity

Answer 2

You could certainly write a XUL application with Mozilla (run it with Firefox, Xulrunner etc) which scripts a web browser. Javascript is normally used for such tasks.

What I've found is tricky is suppressing all the kinds of dialogue boxes which the browser would otherwise create - you effectively have to override the behaviour of the XPCOM server classes which are invoked for each type of dialogue, and there are a lot of different ones (for example, if your site decides to redirect to a https site with an expired certificate).

Of course you should NOT use such a mechanism to violate any site's policy on use by robots. Normally you should never submit a form with a robot.