Fetching or Deleting Entity from Google App Engine DB with Unicode Property Name

Question

I have an Expando model kind in my App Engine datastore and I'm setting many arbitrary property names. I didn't consider that I couldn't store Unicode property names, and now I'm in a troubling situation where any attempt to fetch entities of this kind, or even deleting them to get rid of the offender get the following error:

Traceback (most recent call last):
  File "/base/data/home/apps/APP/1-05.335746938130078870/console/app/models/console.py", line 146, in processSource
    exec bytecode in statement_module.__dict__
  File "<string>", line 1, in <module>
  File "/base/python_lib/versions/1/google/appengine/ext/db/__init__.py", line 1461, in fetch
    return [self._model_class.from_entity(e) for e in raw]
  File "/base/python_lib/versions/1/google/appengine/ext/db/__init__.py", line 1047, in from_entity
    entity_values = cls._load_entity_values(entity)
  File "/base/python_lib/versions/1/google/appengine/ext/db/__init__.py", line 1347, in _load_entity_values
    entity_values[str(key)] = value
UnicodeEncodeError: 'ascii' codec can't encode character u'\xe1' in position 5: ordinal not in range(128)

Now I've changed my code to encode these property names to ascii in advance of saving them, but it's too late! Because there is at least one entity with a Unicode-typed property name, it is impossible for me to fetch the entity or delete it, and this means that the Datastore viewer is returning a "A server error has occurred." error, so I can't even see where the problem is.

The only think I can think of is forking the db module and change the str() method, but it would just be a lot easier for me to find a way to wipe this Model, or even better, to be able to delete the offending entity (or entities) without getting rid of all of the valid ones.

Answer 1

The comment I left ended up working.

Here's the solution (assuming there are under 1000 of such entities):

entities = db.GqlQuery("SELECT __key__ FROM ExpandoModel").fetch(1000)

for e in entities:

  try: db.get(e)
  except UnicodeEncodeError: db.delete(e)

Answer 2

For future reference, you can also import and use the google.appengine.api.datastore module, which provides a lower-level, dict-based interface to the datastore.

You might also want to file a bug about this.