How to delete all datastore in Google App Engine?

Question

Does anyone know how to delete all datastore in Google App Engine?

Answer 1

If you're talking about the live datastore, open the dashboard for your app (login on appengine) then datastore --> dataviewer, select all the rows for the table you want to delete and hit the delete button (you'll have to do this for all your tables). You can do the same programmatically through the remote_api (but I never used it.)

If you're talking about the development datastore, you'll just have to delete the following file: "./WEB-INF/appengine-generated/local_db.bin". The file will be generated for you again next time you run the development server and you'll have a clear db.

Make sure to clean your project afterwards.

This is one of the little gotchas that come in handy when you start playing with the Google Application Engine. You'll find yourself persisting objects into the datastore then changing the JDO object model for your persistable entities ending up with a obsolete data that'll make your app crash all over the place.

Answer 2

You can do it using the web interface. Login into your account, navigate with links on the left hand side. In Data Store management you have options to modify and delete data. Use respective options.

Answer 3

If you have a significant amount of data, you need to use a script to delete it. You can use remote_api to clear the datastore from the client side in a straightforward manner, though.

Answer 4

The best approach is the remote API method as suggested by Nick, he's an App Engine engineer from Google, so trust him.

It's not that difficult to do, and the latest 1.2.5 SDK provides the remote_shell_api.py out of the shelf. So go to download the new SDK. Then follow the steps:

• connect remote server in your commandline: remote_shell_api.py yourapp /remote_api The shell will ask for your login info, and if authorized, will make a Python shell for you. You need setup url handler for /remote_api in your app.yaml

• fetch the entities you'd like to delete, the code looks something like:

    from models import Entry
    query = Entry.all()
    entries =query.fetch(1000)
    db.delete(entries)
    \# This could bulk delete 1000 entities a time

Answer 5

Source

I got this from http://code.google.com/appengine/articles/remote_api.html.

Create the Interactive Console

First, you need to define an interactive appenginge console. So, create a file called appengine_console.py and enter this:

#!/usr/bin/python
import code
import getpass
import sys

# These are for my OSX installation. Change it to match your google_appengine paths. sys.path.append("/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine")
sys.path.append("/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/lib/yaml/lib")

from google.appengine.ext.remote_api import remote_api_stub
from google.appengine.ext import db

def auth_func():
  return raw_input('Username:'), getpass.getpass('Password:')

if len(sys.argv) < 2:
  print "Usage: %s app_id [host]" % (sys.argv[0],)
app_id = sys.argv[1]
if len(sys.argv) > 2:
  host = sys.argv[2]
else:
  host = '%s.appspot.com' % app_id

remote_api_stub.ConfigureRemoteDatastore(app_id, '/remote_api', auth_func, host)

code.interact('App Engine interactive console for %s' % (app_id,), None, locals())

Create the Mapper base class

Once that's in place, create this Mapper class. I just created a new file called utils.py and threw this:

class Mapper(object):
  # Subclasses should replace this with a model class (eg, model.Person).
  KIND = None

  # Subclasses can replace this with a list of (property, value) tuples to filter by.
  FILTERS = []

  def map(self, entity):
    """Updates a single entity.

    Implementers should return a tuple containing two iterables (to_update, to_delete).
    """
    return ([], [])

  def get_query(self):
    """Returns a query over the specified kind, with any appropriate filters applied."""
    q = self.KIND.all()
    for prop, value in self.FILTERS:
      q.filter("%s =" % prop, value)
    q.order("__key__")
    return q

  def run(self, batch_size=100):
    """Executes the map procedure over all matching entities."""
    q = self.get_query()
    entities = q.fetch(batch_size)
    while entities:
      to_put = []
      to_delete = []
      for entity in entities:
        map_updates, map_deletes = self.map(entity)
        to_put.extend(map_updates)
        to_delete.extend(map_deletes)
      if to_put:
        db.put(to_put)
      if to_delete:
        db.delete(to_delete)
      q = self.get_query()
      q.filter("__key__ >", entities[-1].key())
      entities = q.fetch(batch_size)

Mapper is supposed to be just an abstract class that allows you to iterate over every entity of a given kind, be it to extract their data, or to modify them and store the updated entities back to the datastore.

Run with it!

Now, start your appengine interactive console:

$python appengine_console.py <app_id_here>

That should start the interactive console. In it create a subclass of Model:

from utils import Mapper
# import your model class here 
class MyModelDeleter(Mapper):
    KIND = <model_name_here>

    def map(self, entity):
        return ([], [entity])

And, finally, run it (from you interactive console): mapper = MyModelDeleter() mapper.run()

That's it!

Answer 6

This is question How to do this in JAVA??

I need to do bulk delete My data store is full And obviously I cant delete it in 20 20 chunks.

Answer 7

If you have a lot of data, using the web interface could be time consuming. The App Engine Launcher utility lets you delete everything in one go with the 'Clear datastore on launch' checkbox. This utility is now available for both Windows and Mac (Python framework).

Answer 8

I've created an add-in panel that can be used with your deployed App Engine apps. It lists the kinds that are present in the datastore in a dropdown, and you can click a button to schedule "tasks" that delete all entities of a specific kind or simply everything. You can download it here:
http://code.google.com/p/jobfeed/wiki/Nuke

Answer 9

This answer just expands on the one given by Nick.

Here is the script I use to clear the dev_appserver datastore. (This way there is no need to restart it with the -c switch.) It uses the remote_api, which must be configured.

import sys

sys.path.append('C:/Program Files (x86)/Google/google_appengine')
sys.path.append('C:/Program Files (x86)/Google/google_appengine/lib/yaml/lib')

from google.appengine.ext.remote_api import remote_api_stub
from google.appengine.ext import db

remote_api_stub.ConfigureRemoteDatastore(
        None, '/remote_api', lambda: ('_', '_'), 'localhost:8080')

query = db.Query(None, keys_only=True)

count = 0
while True:
    keys = query.fetch(1000)
    if not keys:
        break
    count += len(keys)
    db.delete(keys)

print '%s entities deleted' % count

To modify this script for on a production app, you'd need to change the remote_api_stub.ConfigRemoteDatastore parameters. See this article for instructions. If you're not using Windows 7 64-bit, you'll also have to change the sys.path set up.

Answer 10

The fastest and efficient way to handle bulk delete on Datastore is by using the new mapper API announced on the latest Google I/O.

If your language of choice is Python, you just have to register your mapper in a mapreduce.yaml file and define a function like this:

from mapreduce import operation as op
def process(entity):
 yield op.db.Delete(entity)

On Java you should have a look to this article that suggests a function like this:

@Override
public void map(Key key, Entity value, Context context) {
    log.info("Adding key to deletion pool: " + key);
    DatastoreMutationPool mutationPool = this.getAppEngineContext(context)
            .getMutationPool();
    mutationPool.delete(value.getKey());
}

Answer 11

Also, you can clear the development server datastore when you run the server:

/path/to/google_appengine/dev_appserver.py --clear_datastore myappname/

http://code.google.com/appengine/docs/python/gettingstarted/usingdatastore.html

Answer 12

I was so frustrated about existing solutions for deleting all data in the live datastore that I created a small GAE app that can delete quite some amount of data within its 30 seconds.

How to install etc: http://code.google.com/p/xydra/wiki/GaeMyAdmin