appengine, urlfetch, and the content-length header

Question

I have a Google Appengine app requesting pages from another server using urllib2 POSTs. I recently enabled gzip compression on the other server running Apache2, and the Appengine page requests started failing on key-error, indicating 'content-length' is not in the headers.

I am not explicitly declaring gzip as an accepted encoding in my requests from Appengine, but it is possible Appengine is adding that header. Googling has not turned up any clear indication that Appengine's urlfetch implicitly adds a header to accept gzip encoding.

Apache2, if I recall correctly, omits content-length headers when the response is compressed, but that should not affect non-compressed responses from the same server.

Does anybody have any insight as to what is happening, why the content-length header is being omitted?

Answer 1

When POSTing you must send the "Content-Length" with your request, even if it is 0. Do you have content in your POST? Is the "Content-Length" header being sent with it?

You can see the HTTP traffic by turning on debugging:

>>> import urllib2
>>> h=urllib2.HTTPHandler(debuglevel=1)
>>> opener = urllib2.build_opener(h) 
>>> urllib2.install_opener(opener)
>>> o = opener.urlopen('www.google.com')

Answer 2

According to this thread: http://groups.google.com/group/google-appengine-java/browse%5Fthread/thread/5c5f2a7e2d2beadc?pli=1) on an Appengine Java newsgroup, Google does generally set the 'Accept-Encoding: gzip' header on urlfetch requests, and then decompresses (ungzips) the input before handing the data to the script.

So apparently, Appengine adds an accept-encoding: gzip header implicitly on the requests way out to the internet, and decompresses the response, but does not insert a content-length into the headers for the decompressed data size. So if the outside server will provide gzipped responses, the net result to the Appengine script (after all the pre- and post- processing behavior by Appengine described above) is the loss of the content-length header.