Controlling a browser from Python

Question

I am looking for a way to control a browser from Python, i.e. fill out form fields and submit them, possibly call JS functions. I've looked around a bit, but as far as I could see PyWebKitGtk only lets you show the browser as a GUI element, not interface with it.

Is there a way to do this easily? I wrote my program logic in Python, and I would hate to port it to JS. Besides that, even if I'd use pure JS "bookmarklets", those wouldn't be able to read/write to my local filesystem, would they?

Also, some of the content on the page is generated using AJAX, so I'm looking for a solution where javascript runs as normal.

P.S. to quell your suspicions, I'm not trying to automatically fill out forum account creation forms or something similarly spammious, though the task is technically similar. I need to crawl/scrape sites for my research project.

EDIT: IEC looked promising, but I'm working on a Mac.

Answer 1

Why not just use Python to simulate a browser, parsing the HTML and building the appropriate HTTP requests as necessary?

A quick google yields the following:

• http://www.packtpub.com/article/web-scraping-with-python
• http://stackoverflow.com/questions/1419357/scraping-simulate-browsing-help
• http://stockrt.github.com/p/emulating-a-browser-in-python-with-mechanize/
• http://code.activestate.com/recipes/572202-web-browser-emulator/

Answer 2

You might try something like Selenium, which is an application to script browsers through various languages including Python - it's meant for cross-browser testing, but might do what you want.

Answer 3

use mechanize module