Some ideas:
1) App engine can serve more than one request at a time. Try it out - app engine will probably spin up more than one instance of your app => multiple requests can be done at once. With request times so long though, I wouldn't expect it to scale much though (they recommend request/response time in under 1 second - see this link).
2) If you wanted to return quickly to the user, you could enqueue to the task queue as you suggested. Then have the user's webpage (via meta tag http-equiv or JavaScript perhaps) poll the server every couple seconds to see if the page is ready.
3) If the generated page may be needed again, you should consider memcaching it to try to save the effort of generating it again. With load times of 10 seconds, you might even consider storing them in the datastore for a little while (if caching is appropriate for your app).
Here's a very basic example of how you might do this:
import hashlib
from google.appengine.api.labs import taskqueue
from google.appengine.ext import db, webapp
from google.appengine.ext.webapp.util import run_wsgi_app
class BuiltPage(db.Model):
html = db.TextProperty()
class PageBuilder(webapp.RequestHandler):
"""Handler called by the task queue to build the page."""
def post(self):
key = self.request.get('key')
url = self.request.get('url')
import time
time.sleep(5) # pretend it takes a while to build the page
html = "you asked for %s" % url # do real stuff to build the page here ...
BuiltPage(key_name=key, html=html).put() # should check for errors ...
def html_to_redir_to_built_page(hexkey):
"Page to show while we wait. Auto-refreshes until we get the built page."""
new_url = '/get_built_page?k=' + hexkey
refresh_tag = '<meta http-equiv="refresh" content="2;%s"/>' % new_url
return '<html><head>%s</head><body>please wait</body></html>' % refresh_tag
class BuildPageForURL(webapp.RequestHandler):
"""Handles requests by a user to build the page for the request URL."""
def get(self):
url = self.request.get('url')
key = hashlib.md5(url).hexdigest()
# optimization: check datastore to see if it was already generated?
taskqueue.add(url='/buildit', params=dict(key=key, url=url))
self.redirect('/get_built_page?k=' + key)
class GetBuiltPage(webapp.RequestHandler):
"""Returns the built page if it is ready, otherwise returns a page which will retry later"""
def get(self):
key = self.request.get('k')
bp = BuiltPage.get_by_key_name(key)
if bp:
self.response.out.write(bp.html)
# maybe cleanup if you know this is a 1-time request: bp.delete()
else:
self.response.out.write(html_to_redir_to_built_page(key))
application = webapp.WSGIApplication([('/', BuildPageForURL),
('/buildit', PageBuilder),
('/get_built_page', GetBuiltPage)])
def main(): run_wsgi_app(application)
if __name__ == '__main__': main()