Duplicate key-value pairs returned by memcached

Question

hi - We are using a cluster of memcached servers for caching purpose, in a Django(Python) production, having tried both cmemcache and python-memcache as the API. The problem is under high concurrency, we started to have duplicate key-value pairs, that is to say we are having multi values for a single key. Is there anyone having had the same similar situation and what is the kill?

Since the memcached servers themselves are not communicating with each to maintain the singularity of the key-value pair, this task is left to the client library, so we are trying to understand further how cmemcache and python-memcache works.

Answer 1

Which is a bigger problem: having redundant data, or getting more than one result back for a query?

If the former is the problem, then you've got a nasty concurrency issue to solve on your hands.

If the latter is the problem, however, why not just give each host that's storing values in memcache a unique identifier, and prepend that to any key?

Answer 2

The client is supposed to figure out which memcache server a value lives on based on its key. If you're using two different clients (or two different configurations of the same client) they might be using different algorithms to map a key to a server, thereby sending values for the same key to two different servers.

You might want to switch to pylibmc or python-libmemcached. When I reviewed python memcache clients last year, both of the clients you mentioned were either retired or broken or both.