How to save a whole web page with something block in it

Question

Hi everyone,

I want to save a web page. I use python urllib to parse the web page. But I find the saved file, where some content is missing. The missing part is block from the source web page, such as this part <div style="display: block;" id="GeneInts">...</div>. I don't know how to parse a whole page without something block in it. Could you help me figure it out? Thank you!

This is my program

url = 'http://receptome.stanford.edu/hpmr/SearchDB/getGenePage.asp?Param=4502931&ProtId=1&ProtType=Receptor'
f = urllib.urlretrieve(url,'test.html') 

Answer 1

That page generates a great deal of its content with JavaScript executed at load-time, including, I think, the part you're trying to extract. You need a screen-scraper that can run JavaScript and then save out the modified DOM. I don't know where you get one of those.

Answer 2

Whenever I need to let Javascript operate on a page before I can scrape it, the first thing I always turn to is SeleniumRC -- while it's mainly designed for purposes of testing, I've never found a better tool for this challenging task. For the "using it from Python" part, see here and links therefrom.