How to deal with deflated response by urllib2?

Question

I currently use following code to decompress gzipped response by urllib2:

opener = urllib2.build_opener()
response = opener.open(req)
data = response.read()
if response.headers.get('content-encoding', '') == 'gzip':
    data = StringIO.StringIO(data)
    gzipper = gzip.GzipFile(fileobj=data)
    html = gzipper.read()

Does it handle deflated response too or do I need to write seperate code to handle deflated response?

Answer 1

You need a slightly different approach, see this SO thread.

Answer 2

To answer from above comment, the HTTP spec (http://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html#sec14.3) says:

If no Accept-Encoding field is present in a request, the server MAY assume that the client will accept any content coding. In this case, if "identity" is one of the available content-codings, then the server SHOULD use the "identity" content-coding, unless it has additional information that a different content-coding is meaningful to the client.

I take that to mean it should use identity. I've never seen a server that doesn't.

Answer 3

There is a better way outlined at:

• http://rationalpie.wordpress.com/2010/06/02/python-streaming-gzip-decompression/

The author explains how to decompress chunk by chunk, rather than all at once in memory. This is the preferred method when larger files are involved.

Also found this helpful site for testing:

• h__//carsten.codimi.de/gzip.yaws/