Tools to work with App Engine data dumps

Question

Using the bulkloader.py utility you can download all data from your application's Datastore.

It is not obvious how the data is stored, however. From the looks of it, you get a SQLite file with all data in binary format in a single table:

sqlite> .tables
bulkloader_database_signature  result  

sqlite> .schema result
CREATE TABLE result (
  id BLOB primary key,
  value BLOB not null,
  sort_key BLOB);

Are there any tools to work with this data?

Answer 1

Okay, no tools yet. Just to share what I have found so far:

The binary data stored in the column value is in Google Protocol Buffer format. Every row represents one entity.

Using the Java SDK, you can decode the data:

 OnestoreEntity.EntityProto m = new OnestoreEntity.EntityProto();
 m.parseFrom(data);
 Entity entity = EntityTranslator.createFromPb(m);

 System.out.println(entity.getAppId());
 System.out.println(entity.getKind());
 System.out.println(entity.getKey());

 // the properties (a Map)
 System.out.println(entity.getProperties());