views:

31

answers:

1

How come I hit this webpage, I get HTML text:

http://itunes.apple.com/us/app/mobile/id381057839

But when I hit this webpage, I get garbled junk?

http://itunes.apple.com/us/app/mobile/id375562663

I use the same download() function in python, which is here:

def download(source_url):
    try:
        socket.setdefaulttimeout(10)
        agent = "Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US; rv:1.9.2.10) Gecko/20100914 AlexaToolbar/alxf-1.54 Firefox/3.6.10 GTB7.1"
        ree = urllib2.Request(source_url)
        ree.add_header('User-Agent',agent)
        ree.add_header("Accept","text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8")
        ree.add_header("Accept-Language","en-us,en;q=0.5")
        ree.add_header("Accept-Charset","ISO-8859-1,utf-8;q=0.7,*;q=0.7")
        ree.add_header("Accept-Encoding","gzip,deflate")
        ree.add_header("Host","itunes.apple.com")
        resp = urllib2.urlopen(ree)
        htmlSource = resp.read()
        return htmlSource
    except Exception, e:
        print e
+1  A: 

Solved. It was compression issue.

def download(source_url):
    try:
        socket.setdefaulttimeout(10)
        agents = ['Mozilla/4.0 (compatible; MSIE 5.5; Windows NT 5.0)','Mozilla/4.0 (compatible; MSIE 7.0b; Windows NT 5.1)','Microsoft Internet Explorer/4.0b1 (Windows 95)','Opera/8.00 (Windows NT 5.1; U; en)']
        ree = urllib2.Request(source_url)
        ree.add_header('User-Agent',random.choice(agents))
        ree.add_header('Accept-encoding', 'gzip')
        opener = urllib2.build_opener()
        h = opener.open(ree).read()
        import StringIO
        import gzip

        compressedstream = StringIO.StringIO(h)
        gzipper = gzip.GzipFile(fileobj=compressedstream)
        data = gzipper.read()
        return data

    except Exception, e:
        print e
        return ""
TIMEX
Hmm, maybe you could check if the resource is gzipped, unless you're sure you'll always get gzipped response back.
Piskvor