How to parse a web use javascript to load .html by Python?

Question

Hi,

I'm using Python to parse an auction site.

If I use browser to open this site, it will go to a loading page, then jump to the search result page automatically.

If I use urllib2 to open the webpage, the read() method only return the loading page.

Is there any python package could wait until all contents are loaded then read() method return all results?

Thanks.

Answer 1

How does the search page work? If it loads anything using Ajax, you could do some basic reverse engineering and find the URLs involved using Firebug's Net panel or Wireshark and then use urllib2 to load those.

If it's more complicated than that, you could simulate the actions JS performs manually without loading and interpreting JavaScript. It all depends on how the search page works.

Lastly, I know there are ways to run scripting on pages without a browser, since that's what some functional testing suites do, but my guess is that this could be the most complicated approach.

Answer 2

After tracing for the auction web source code, I found that it uses .php to create loading page and redirect to result page. Reverse engineering to find the ture URLs is not working because it's the same URL as loading page.

And @Manoj Govindan, I've tried Mechanize, but even if I add

br.set_handle_refresh(True)
br.set_handle_redirect(True)

it still read the loading page.

After hours of searching on www, I found a possible solution : using pywin32

import win32com.client
import time

url = 'http://search.ruten.com.tw/search/s000.php?searchfrom=headbar&k=halo+reach'
ie = win32com.client.Dispatch("InternetExplorer.Application")
ie.Visible = 0
ie.Navigate(url)

while 1:
  state = ie.ReadyState
  if state == 4:
    break
    time.sleep(1)

print ie.Document.body.innerHTML

However this only works on win32 platform, I'm looking for a cross platform solutoin.

If anyone know how to deal this, please tell me.