Retrieving information with Python's urllib from a page that is done via __doPostBack()?

Question

I'm trying to parse a page that has different sections that are loaded with a Javascript __doPostBack() function.

An example of a link is: javascript:__doPostBack('ctl00$cphMain$ucOemSchPicker$dlSch$ctl03$btnSch','')

As soon as this is clicked, the browser doesn't fetch a new URL but a section of webpage is updated to reflect new information.

What would I pass into a urllib function to complete the operation?

Answer 1

javascript:__doPostBack('...

(Urgh. That's a sad and nasty approach.)

A simple general-purpose approach for finding URLs whose logic is buried in JavaScript is to run the page normally, with a network debugger on (eg. Firebug's ‘Net’ tab, or Fiddler). By monitoring the request made when you click, you can see what URL and what POST request body parameters are to be passed.

You'll need to use the data argument of urlopen to send POST request bodies.