Dealing with db.Timeout on Google App Engine

Question

I'm testing my application (on Google App Engine live servers) and the way I've written it I have about 40 db.GqlQuery() statements in my code (mostly part of classes).

I keep getting db.Timeout very often though.

How do I deal with this? I was going to surround all my queries with really brutal code like this:

  querySucceeded = False
  while not querySucceeded :
    try :
      result = db.GqlQuery( """xxx""" ).get()
      querySucceeded = True #only get here if above line doesn't raise exc
    except :
      querySucceeded = False

Is this ok? Do you agree? What's a better way to deal with db.Timeouts?

Edit:

I now use this for any get queries

""" Query gets single result """
def queryGet( gql ) :
  querySucceeded = False
  while not querySucceeded :
    try :
      result = db.GqlQuery( gql ).get()
      querySucceeded = True #only get here if above line doesn't raise
    except :
      querySucceeded = False

  return result

I have similar functions for fetch and count.

Answer 1

Queries will occasionally fail. You can either show an error message to the user, or retry, as you're doing above. If you retry, however, you should use thread.sleep to add increasing amounts of delay (starting at, say, 50ms) on each retry - retries are more likely to succeed if they're not retried as fast as possible.

40 queries per request is a lot, though. You should consider refactoring your code - it must be possible to eliminate most of those!

Answer 2

See the new ROTModel in GAE-Utilities. The second discussion below shows how it does retries. It's a subclass of db.Model, so your classes can inherit from ROTModel instead, and take advantage of it's retries.

http://code.google.com/p/gaeutilities

http://groups.google.com/group/google-appengine/browse_thread/thread/ac51cc32196d62f8/aa6ccd47f217cb9a?lnk=gst&q=timeout#aa6ccd47f217cb9a

Answer 3

Here's a decorator to retry on db.Timeout, adapted from one from Kay framework:

import logging, time
from google.appengine.ext import db

def retry_on_timeout(retries=3, interval=1.0, exponent=2.0):
    """A decorator to retry a given function performing db operations."""
    def _decorator(func):
        def _wrapper(*args, **kwargs):
            count = 0
            while True:
                try:
                    return func(*args, **kwargs)
                except db.Timeout, e:
                    logging.debug(e)
                    if count >= retries:
                        raise e
                    else:
                        sleep_time = (exponent ** count) * interval
                        logging.warning("Retrying function %r in %d secs" %
                            (func, sleep_time))
                        time.sleep(sleep_time)
                        count += 1

        return _wrapper

    return _decorator

To use it, simply decorate any function that performs db operations and you'd like to do retries:

@retry_on_timeout()
def do_the_stuff(models):
    return db.put(models)

Answer 4

Take a look at this this python Autoretry Datastore Timeouts recipe. Similar to moraes answer, but you only need to call it once at initialization time, rather than decorate functions that perform datastore operations.

http://appengine-cookbook.appspot.com/recipe/autoretry-datastore-timeouts