Google App Engine database inconsistency

Question

Im writing a twitter app in Google App Engine. It accepts commands as Direct Messages, so i have setup a third party cronjob service to invoke a handler that processes DMs at regular intervals. I have a Model 'Info' that has just one entry, it stores some common data which are used in many places in the App(in this case, the time when the messages were processed recently). The general pattern of my handler is like this:

class Info(db.Model):
    msg_polled = db.DateTimeProperty(auto_now_add = True)
    .... More Properties ....

    @classmethod
    def get_info(cls):
        info = cls.all().get()
        if not info:
            info = cls()
            info.put()
        return info
---------------------------------------------------------
info = Info.get_info()
msgs = api.GetDirectMessages(since = info.msg_polled)
if not msgs:
    return
logging.info('Processing Messages since %s ' % str(info.msg_polled))
for msg in msgs:

    ...process commands...

    logging.info('Processed Message :- @%s : %s' % (msg.sender_screen_name, msg.text))

info.msg_polled = datetime.datetime.now()
info.put()

But sometimes i get logs like this :

I 03-30 07:50AM 10.973
Processing Messages since Sun, 29 Mar 2009 11:41:59 GMT 
I 03-30 07:50AM 11.122
Processed Message :- @foo : Foo_Bar
-------------------------------------------------------
I 03-30 07:46AM 08.014
Processing Messages since Sun, 29 Mar 2009 11:41:59 GMT 
I 03-30 07:46AM 08.130
Processed Message :- @foo : Foo_Bar

Here, it seems that info is not getting commited to the database. The message is processed multiple number of times, sometimes upto 10+ times before the msg_polled value changes. But i am not getting any Datastore exceptions. This happens only once in a while.

Any help is appreciated.

Answer 1

The Google AppEngine datastore uses BigTable under the covers, which is a distributed database system. Because of this, updates might not be visible immediately since the new data has not yet reached every distributed table (Amazon calls this "eventual consistency" in their SimpleDB). You should be fine after a few minutes.