Memcached - how to deal with adding/deploying servers

Question

Hi everybody,

How do you handle replacing/adding/removing memcached nodes in your production applications? I will have a number of applications that are cloned and customized due to each customers need running on one and same webserver, so i'll guess that there will be a day when some of the nodes will be changed.

Here's how memcached is populated by normal:

$m = new Memcached();

$servers = array(
    array('mem1.domain.com', 11211, 33),
    array('mem2.domain.com', 11211, 67)
);
$m->addServers($servers);

My initial idea, is to make the $servers array to be populated from the database, also cached, but file-based, done once a day or something, with the option to force an update on next run of the function that holds the $addservers call. However, I am guessing that this might add some additional overhead since disks are quite slow storage...

What do you think?

Answer 1

It's always better to define things like that in the source. Granted, you could use parse_ini_file() or something like that to read a configuration file. Really though, it belongs in the source. But if defined in an INI file, there aren't many security concerns such as code injection and the likes as opposed to defining your entries in PHP.

Since there are a few applications that might be reading the same INI file, you can put it in a common location that all applications have access to (using group permissions, for example). Actually doing that isn't really recommended.

Answer 2

Depending on your requirements I'd use a list that supports auto-eviction after some TTL (passive mode) or some sort of listener pattern if you want this to be updated immediately (active mode). Wherever you keep your primary source (properties file, database, etc.) is not really important as long as it is reachable when necessary and your TTL is big enough (as you mentioned in your comment).

For passive approach you could use reuse some other local caching library that supports TTL (e.g. EHCache in Java) by putting your list of servers into the cache under some known key. For active approach, for example, Java has java.util.EventListener. It is (un)fortunately many years since I've done anything in PHP so can't advise on that.

I would strongly advise against putting configuration data into the code. In the world I'm living hardcoding is a grave sin.

You might also want to have a look at how last.fm is using consistent hashing algorithm for being able to add/remove servers to/from the memcached pool without causing a complete remap of all keys.

Answer 3

Use a config file - whether its from parse_ini_file or some other config parsing scheme. If you are worried about parsing time, you could put those configs in the php ini load path http://brian.moonspot.net/using-ini-files-for-php-application-settings - this pre-parses all of the vars you need, but you'll need to restart apache to load any changes.

Once the files are read enough (as long as you dont have too much thrashing on your machine) the config file will be in memory mapped cache anyway, so it'll be really fast. You can optimize by using the php serialized form which loads fast, and even optimize further by using the APC user cache.

Lastly, use the newest memcache client libraries - they use the consistent hashing algorithms now, and it'll help adding/removing individual servers.

Answer 4

As far as I udnerstood, the main concern here is how to cache the configuration keeping it mostly up-to-date when it changes.

Well, the best option here is obviously storing it in a database and refreshing from DB, say, every 15 seconds. Having big load on the application, querying the DB once a 15 secs won't change anything at all. Loading the data from DB itself is quite quick as you need just a couple of fields.

Another option that might work here – use a separate memcached :) Seriously, just cache the configuration loaded from DB there, purging that cache key on configuration update.

To sum up: any time-based expiration scheme would work. The simpliest solution – maintain it on your own (store last refresh time and check it on every function call); a bit more advanced – use something like memcached.

update: DB is the best solution as it scales pretty well: you don't have to copy configs across 20 servers on every configuration update.