Python-memcached is the official supported memcached driver for Django.
Does it support
- Consistent hashing
- Binary protocol
If it does, how do I use those features within Django? I couldn't find any documentation.
Python-memcached is the official supported memcached driver for Django.
Does it support
If it does, how do I use those features within Django? I couldn't find any documentation.
Looking at the _get_server
method on python-memcached v1.45, it seems it doesn't use consistent hashing, but a simple hash % len(buckets)
.
Same goes for binary protocol, python-memcache uses, as far as I can see in the source, only text commands.
You might be able to use this: http://amix.dk/blog/post/19370
It encapsulates python-memcache's Client class so keys are distributed using consistent hashing.
EDIT- I'm digging in python-memcached
1.4.5 source code, and it looks like it might actually support consistent hashing.
Relevant code:
from binascii import crc32 # zlib version is not cross-platform
def cmemcache_hash(key):
return((((crc32(key) & 0xffffffff) >> 16) & 0x7fff) or 1)
serverHashFunction = cmemcache_hash
-- SNIP --
def _get_server(self, key):
if isinstance(key, tuple):
serverhash, key = key
else:
serverhash = serverHashFunction(key)
for i in range(Client._SERVER_RETRIES):
server = self.buckets[serverhash % len(self.buckets)]
if server.connect():
#print "(using server %s)" % server,
return server, key
serverhash = serverHashFunction(str(serverhash) + str(i))
return None, None
Based on this code, it looks like it does implement the algorithm, unless cmemcache_hash
is not a meaningful name and that is not the real algorithm. (the now retired cmemcache does consistent hashing)
But I think the OP is referring to more "resilient" consistent hashing, e.g. libketama. I don't think there's a drop in solution out there for that out there, looks like you need to roll up your sleeves compile/install a more advanced memcached lib like pylibmc, and write a custom Django backend that uses that instead of python-memcached.
Anyhow, in either case, some remapping of keys will occur when you add/remove buckets to the pool (even with libketama, just less than with the other algorithms)