Storing complex object on Datastore with Pickle, any faster alternatives?

Question

To store a complex object to Datastore i'm currently using this approach:

class PickleProperty(db.Property): 
  data_type = db.Blob
  def get_value_for_datastore(self, model_instance):
    value = self.__get__(model_instance, model_instance.__class__)
    if value is not None:
      return db.Blob(pickle.dumps(value))
  def make_value_from_datastore(self, value):
    if value is not None:
      return pickle.loads(str(value))

class ComplexModel(db.Model):
      data = PickleProperty()

As you know, Google App Engine does not have the more efficient cPickle module; this results in a very slow operation.

Any better strategy?

Answer 1

The docs says that model_to_protobuf wants a Model instance, and if your object already is a Model you could just as well make a ReferenceProperty pointing to that object directly. I'd suggest you first benchmark pickling with decently realistic data to find out exactly how slow pickling/unpickling is. It might not be too slow, but it might be good to actually know that before committing to work-intensive fixes, premature optimization being evil and all that (as you probably know..).

But if it actually is slow I'd suggest you try to model the objects that needs serializing in a less capable format (and hence probably faster), for example JSON (which would be practical, since you already have JSON capabilities in app engine present (import it as django.utils.simplejson).

If it's possible to serialize the objects as Model instances I suggest you do that. If you don't know all attributes until runtime, there's always Expando models.