Python decorators and class methods and evaluation -- django memoize

Question

I have a working memoize decorator which uses Django's cache backend to remember the result of a function for a certain amount of time. I am specifically applying this to a class method.

My decorator looks like:

def memoize(prefix='mysite', timeout=300, keygenfunc=None):
# MUST SPECIFY A KEYGENFUNC(args, kwargs) WHICH MUST RETURN A STRING
def funcwrap(meth):

  def keymaker(*args, **kwargs):
    key = prefix + '___' + meth.func_name + '___' + keygenfunc(args, kwargs)
    return key

  def invalidate(*args, **kwargs):
    key = keymaker(*args, **kwargs)
    cache.set(key, None, 1)

  def newfunc(*args, **kwargs):
    # construct key
    key = keymaker(*args, **kwargs)

    # is in cache?
    rv = cache.get(key)

    if rv is None:
      # cache miss
      rv = meth(*args, **kwargs)
      cache.set(key, rv, timeout)

    return rv

  newfunc.invalidate = invalidate
  return newfunc
return funcwrap

I am using this on a class method, so something like:

class StorageUnit(models.Model):
  @memoize(timeout=60*180, keygenfunc=lambda x,y: str(x[0].id))
  def someBigCalculation(self):
    ...
    return result

The actual memoization process works perfectly! That is, a call to

myStorageUnitInstance.someBigCalculation()

properly uses the cache. OK, cool!

My problem is when I try to manually invalidate the entry for a specific instance, where I want to be able to run

myStorageUnitInstance.someBigCalculation.invalidate()

However, this doesn't work, because "self" doesn't get passed in and therefore the key doesn't get made. I get a "IndexError: tuple index out of range" error pointing to my lambda function as shown earlier.

Of course, I can successfully call:

myStorageUnitInstance.someBigCalculation.invalidate(myStorageUnitInstance)

and this works perfectly. But it "feels" redundant when I'm already referencing a specific instance. How can I make Python treat this as an instance-bound method and therefore properly fill in the "self" variable?

Answer 1

Descriptors must always be set on the class, not on the instance (see the how-to guide for all details). Of course, in this case you're not even setting it on the instance, but rather on another function (and fetching it as an attribute of a bound method). I think that the only way to use the syntax you want is to make funcwrap an instance of a custom class (which class must be a descriptor class, of course, i.e., define the appropriate __get__ method, just like functions intrinsically do). Then invalidate can be a method of that class (or, perhaps better, the other custom class whose instance is the "bound-method-like substance" produced by the formerly mentioned descriptor class's __get__ method), and eventually reach the im_self (that's how it's named in a bound method) that you crave.

A pretty hefty (conceptual and coding;-) price to pay for the minor convenience you seek -- hefty enough that I don't really feel like spending an hour or two developing it completely and testing it. But I hope I've given you clear-enough indications for you to proceed if you're still keen on this, and indeed I'll be glad to clarify and help out if there's anything unclear or something is stumping you along this progress.

Answer 2

While I agree with AlexM, I did have some spare time and thought this would be interesting:

# from django.whereever import cache
class memoize(object):
    def __init__(self,prefix='mysite', timeout=300, keygenfunc=None):
        class memo_descriptor(object):
            def __init__(self,func):
                self.func = func
            def __get__(self,obj,klass=None):
                key = prefix + '___' + self.func.func_name + '___' + keygenfunc(obj)
                class memo(object):
                    def __call__(s,*args,**kwargs):
                        rv = cache.get(key)
                        if rv is None:
                            rv = self.func(obj,*args, **kwargs)
                            cache.set(key, rv, timeout)
                        return rv
                    def invalidate(self):
                        cache.set(key, None, 1)
                return memo()
        self.descriptor = memo_descriptor
    def __call__(self,func):
        return self.descriptor(func)

Note I've changed the keygenfunc signature from (*args,**kwargs) to (instance), as that is how you were using it in your example (and it's impossible to have an someBigCalculation.invalidate method clear the cache in the manner you wanted if you generate a key from the arguments to the method call rather than the object instance).

class StorageUnit(models.Model):
    @memoize(timeout=60*180, keygenfunc=lambda x: str(x.id))
    def someBigCalculation(self):
        return 'big calculation'

There is a lot of stuff going on in that code, so whether it's actually making your life easier is something to consider.