Python: urlopen not downloading the entire site.

Question

Greetings,

I have done:

import urllib

site = urllib.urlopen('http://www.weather.com/weather/today/Temple+TX+76504')
site_data = site.read()
site.close()

but it doesn't compare to viewing the source when loaded in firefox.

I suspected the user agent and did this:

class AppURLopener(urllib.FancyURLopener):
    version = "Mozilla/5.0 (X11; U; Linux i686; zh-CN; rv:1.9.2.8) Gecko/20100722 Ubuntu/10.04 (lucid) Firefox/3.6.8"

urllib._urlopener = AppURLopener()

and downloaded it, but it still doesn't download the whole website.

Can someone please help me do user agent switching, if that is the likely culprit?

Thanks, Narnie

Answer 1

It's more likely that there is an iframe in the code or that javascript is modifying the DOM. If theres an iframe, you'll have to parse the page to get the url for the iframe or just do it manually if it's a one-off. If it's javascript, I hear that selenium-rc is good but have no first hand experience with it.

Answer 2

downloaded page displayed locally may look different from several reasons, like that there are relative links (can be fixed adding e.g. <base href="http://www.weather.com/today/"&gt; into the page head element), or non-functional ajax requests (see http://stackoverflow.com/questions/3076414/ways-to-circumvent-the-same-origin-policy).