is there a pythonic way to try something up to a maximum number of times?

Question

I have a python script which is querying a MySQL server on a shared linux host. For some reason, queries to MySQL often return a "server has gone away" error:

_mysql_exceptions.OperationalError: (2006, 'MySQL server has gone away')

If you try the query again immediately afterwards, it usually succeeds. So, I'd like to know if there's a sensible way in python to try to execute a query, and if it fails, to try again, up to a fixed number of tries. Probably I'd want it to try 5 times before giving up altogether.

Here's the kind of code I have:

conn = MySQLdb.connect(host, user, password, database)
cursor = conn.cursor()

try:
    cursor.execute(query)
    rows = cursor.fetchall()
    for row in rows:
        # do something with the data
except MySQLdb.Error, e:
    print "MySQL Error %d: %s" % (e.args[0], e.args[1])

Clearly I could do it by having another attempt in the except clause, but that's incredibly ugly, and I have a feeling there must be a decent way to achieve this.

Thanks,

Ben

Answer 1

How about:

conn = MySQLdb.connect(host, user, password, database)
cursor = conn.cursor()
attempts = 0

while attempts < 3:
    try:
        cursor.execute(query)
        rows = cursor.fetchall()
        for row in rows:
            # do something with the data
        break
    except MySQLdb.Error, e:
        attempts += 1
        print "MySQL Error %d: %s" % (e.args[0], e.args[1])

Answer 2

I'd refactor it like so:

def callee(cursor):
    cursor.execute(query)
    rows = cursor.fetchall()
    for row in rows:
        # do something with the data

def caller(attempt_count=3, wait_interval=20):
    """:param wait_interval: In seconds."""
    conn = MySQLdb.connect(host, user, password, database)
    cursor = conn.cursor()
    for attempt_number in range(attempt_count):
        try:
            callee(cursor)
        except MySQLdb.Error, e:
            logging.warn("MySQL Error %d: %s", e.args[0], e.args[1])
            time.sleep(wait_interval)
        else:
            break

Factoring out the callee function seems to break up the functionality so that it's easy to see the business logic without getting bogged down in the retry code.

Answer 3

Building on Dana's answer, you might want to do this as a decorator:

def retry(howmany):
    def tryIt(func):
        def f():
            attempts = 0
            while attempts < howmany:
                try:
                    return func()
                except:
                    attempts += 1
        return f
    return tryIt

Then...

@retry(5)
the_db_func()

Enhanced version that uses the decorator module

import decorator, time

def retry(howmany, *exception_types, **kwargs):
    timeout = kwargs.get('timeout', 0.0) # seconds
    @decorator.decorator
    def tryIt(func, *fargs, **fkwargs):
        for _ in xrange(howmany):
            try: return func(*fargs, **fkwargs)
            except exception_types or Exception:
                if timeout is not None: time.sleep(timeout)
    return tryIt

Then...

@retry(5, MySQLdb.Error, timeout=0.5)
the_db_func()

To install the decorator module:

$ easy_install decorator

Answer 4

conn = MySQLdb.connect(host, user, password, database)
cursor = conn.cursor()

for i in range(3):
    try:
        cursor.execute(query)
        rows = cursor.fetchall()
        for row in rows:
            # do something with the data
        break
    except MySQLdb.Error, e:
        print "MySQL Error %d: %s" % (e.args[0], e.args[1])

Answer 5

Like S.Lott, I like a flag to check if we're done:

conn = MySQLdb.connect(host, user, password, database)
cursor = conn.cursor()

success = False
attempts = 0

while attempts < 3 and not success:
    try:
        cursor.execute(query)
        rows = cursor.fetchall()
        for row in rows:
            # do something with the data
        success = True 
    except MySQLdb.Error, e:
        print "MySQL Error %d: %s" % (e.args[0], e.args[1])
        attempts += 1