Emptying the datastore in GAE

Question

I know what you're thinking, 'O not that again!', but here we are since Google have not yet provided a simpler method.

I have been using a queue based solution which worked fine:

import datetime from models import *

DELETABLE_MODELS = [Alpha, Beta, AlphaBeta]

def initiate_purge():
    for e in config.DELETABLE_MODELS:
        deferred.defer(delete_entities, e, 'purging', _queue = 'purging')

class NotEmptyException(Exception): pass

def delete_entities(e, queue):
    try:
        q = e.all(keys_only=True)
        db.delete(q.fetch(200))
        ct = q.count(1)
        if ct > 0:
            raise NotEmptyException('there are still entities to be deleted')
        else:
            logging.info('processing %s completed' % queue)
    except Exception, err:
        deferred.defer(delete_entities, e, then, queue, _queue = queue)
        logging.info('processing %s deferred: %s' % (queue, err))

All this does is queue a request to delete some data (once for each class) and then if the queued process either fails or knows there is still some stuff to delete, it re-queues itself.

This beats the heck out of hitting the refresh on a browser for 10 minutes.

However, I'm having trouble deleting AlphaBeta entities, there are always a few left at the end. I think because it contains Reference Properties:

class AlphaBeta(db.Model):
    alpha = db.ReferenceProperty(Alpha, required=True, collection_name='betas')
    beta = db.ReferenceProperty(Beta, required=True, collection_name='alphas')

I have tried deleting the indexes relating to these entity types, but that did not make any difference.

Any advice would be appreciated please.

Answer 1

I don't believe that trying to delete an entity that has references to still-existing entities is really a problem, but you can always rules this out be re-writing your task to delete the entities serially instead of in parallel:

def initiate_purge():
    deferred.defer(delete_entities, Alpha, _queue = 'purging')

def delete_entities(e):
    try:
        q = e.all(keys_only=True)
        db.delete(q.fetch(200))
        ct = q.count(1)
        if ct > 0:
            raise NotEmptyException('there are still entities to be deleted')
        else:
            logging.info('processing completed')
            if type(e) == Alpha:
                logging.info('spawning delete Beta task.')
                deferred.defer(delete_entities, Beta, _queue = 'purging')
            else if type(e) == Beta:
                logging.info('spawning delete AlphaBeta task.')
                deferred.defer(delete_entities, AlphaBeta, _queue = 'purging')
    except Exception, err:
        deferred.defer(delete_entities, e, _queue = 'purging')
        logging.info('processing deferred: %s' % (err))