views:

69

answers:

1

I am starting to use memcache more frequently to avoid having to recalculate things between page requests. When the memcache periodically clears, as it is designed to do, I have to start all over rebuilding various items that I have placed in memcache. What I would like to do is create a very simple model that enables me to periodically save the items that I put into memcache based on the memcache keys that I'm using along with a datetime that is related to the data being memcached. What is the best way to do this?

I'm looking for something like this:

class MemcacheRecord(db.Model):
  key = db.StringProperty(required=True)
  value = #Something that can store whatever memcache can
  validThru = db.DateTimeProperty(required=True)

  def set(self, key, value, validThru):
      #Save a new memcache record
      newMemcacheRecord = MemcacheRecord(key=key, value=value, validThru=validThru)
      ..
      return True # or False

  def get_latest(self, key):
      #Get the memcache record with the most recent validThru datetime
      latestMemcacheRecord = MemcacheRecord.all().order('-validThru').get()
      return {'validThru':latestMemcacheRecord.validThru, 'value':latestMemcachRecord.value}
+1  A: 

There is probably not much benefit in trying to keep track of what you have saved in memcache, because even if you had the list, there is no guarantee that the data has really been preserved.

I think the only way to work with memcache is to try to get data out of it, with the expectation that it might not be there, even if you put it in before. You have to build your algorithms around that.

Thilo
Thilo, Let me elaborate. After I have spent all of the CPU cycles to calculate something, I want to save it to bigtable right before I put it in memcache. Then, as long as the key is in memcache, all pages will pull from there. When the memcache gets cleared, I'll grab the last value saved to bigtable.
Chris
I don't understand. How is this different from trying to load the data from memcache, and if it is not there, go to bigtable? Do you want to speed up the "trying to load from memcache" part?
Thilo
I see. You want to copy everything that you put into memcache into bigtable as well, and are asking for a generic data model (for bigtable) that can accomodate everything that you can store in a memcache record (along with the meta data). Right?
Thilo
Exactly. I'll use the generic data model to refresh memcache entries after they get flushed rather than spend the CPU to rebuild things from scratch.
Chris
I tried to convert dictionaries into json to save in a db.TextProperty field, but json seems to convert any integers in dictionary keys into strings. { 57504L: 31, 57559L: 75} becomes { u'57504': 31, u'57559': 75} when I json.dumps() and then json.loads(). So I'm looking for a more general way to save dictionaries that is compatible with however memcache pickels things.
Chris
Memcache uses pickling - if you want to duplicate the way it does things, you need to pickle your data and use a db.BlobProperty.
Nick Johnson
Thanks Nick. Are there a few short lines of code that you would recommend to pickle a dictionary, save as db.BlobPropoerty, retrieve from BlobProperty, and then un-pickle in GAE?
Chris
@Nick: You'll probably get upvoted and accepted if you moved that answer (with a short sample) into an answer node. Then I can also delete my non-answer.
Thilo