Does Google appengine cache external requests?

Question

I have a very simple application running on appengine that requests a web page every five minutes and parses for a specific piece of data.

Everything works fine except that the response I get back from the external request (using urllib2) doesn't reflect the latest changes to the page. Sometimes it takes a few minutes to get the latest, sometimes over an hour.

Is there a transparent layer of caching that appengine puts in place? Or is there something else I am missing here? I've looked at the caching headers of the requested page and there is no Expires, LastModified or ETags headers sent.

Update: Sometimes, it will get the new version of the page for a number of requests and then randomly later get an old out of date version.

Answer 1

It appears that this is an issue the App Engine team is aware of. The suggested workaround is to set Cache-Control header with max-age in seconds:

result = urlfetch.fetch(url, headers = {'Cache-Control' : 'max-age=240'})

should hopefully work for you.