views:

287

answers:

2

Python-memcached is the official supported memcached driver for Django.

Does it support

  1. Consistent hashing
  2. Binary protocol

If it does, how do I use those features within Django? I couldn't find any documentation.

+2  A: 

Looking at the _get_server method on python-memcached v1.45, it seems it doesn't use consistent hashing, but a simple hash % len(buckets).

Same goes for binary protocol, python-memcache uses, as far as I can see in the source, only text commands.

Clément
Is there any way to make python-memcached use consistent hashing?
Continuation
[hash_ring](http://pypi.python.org/pypi/hash_ring) has a subclass of `memcache.Client` implementing consistent hashing, [from version 1.1](http://amix.dk/blog/post/19370).Then you can implement a [custom cache backend](http://docs.djangoproject.com/en/dev/topics/cache/#using-a-custom-cache-backend) in django, subclassing the original memcached backend so it uses `MemcacheRing`.
Clément
Did you actually verify if it does this or not? Please see my answer above.
Infinity
+2  A: 

You might be able to use this: http://amix.dk/blog/post/19370

It encapsulates python-memcache's Client class so keys are distributed using consistent hashing.

EDIT- I'm digging in python-memcached 1.4.5 source code, and it looks like it might actually support consistent hashing. Relevant code:

from binascii import crc32   # zlib version is not cross-platform
def cmemcache_hash(key):
    return((((crc32(key) & 0xffffffff) >> 16) & 0x7fff) or 1)
serverHashFunction = cmemcache_hash

-- SNIP --

def _get_server(self, key):
    if isinstance(key, tuple):
        serverhash, key = key
    else:
        serverhash = serverHashFunction(key)

    for i in range(Client._SERVER_RETRIES):
        server = self.buckets[serverhash % len(self.buckets)]
        if server.connect():
            #print "(using server %s)" % server,
            return server, key
        serverhash = serverHashFunction(str(serverhash) + str(i))
    return None, None

Based on this code, it looks like it does implement the algorithm, unless cmemcache_hash is not a meaningful name and that is not the real algorithm. (the now retired cmemcache does consistent hashing)

But I think the OP is referring to more "resilient" consistent hashing, e.g. libketama. I don't think there's a drop in solution out there for that out there, looks like you need to roll up your sleeves compile/install a more advanced memcached lib like pylibmc, and write a custom Django backend that uses that instead of python-memcached.

Anyhow, in either case, some remapping of keys will occur when you add/remove buckets to the pool (even with libketama, just less than with the other algorithms)

Infinity
Sorry but this is **not** consistent hashing. `cmemcache_hash` just calculate a plain hash of a key to choose the server that will get the request, and you can clearly see here that the library does this with a simple modulo on the (weighted) server list. Also, `libketama` of `MemcacheRing` are implementing the same algorithm, but `MemcacheRing`, with its implementation of `memcache.Client` is ready for integration.
Clément