Memcached Writes Extremely Slow

Question

I'm running memcached on ec2 instances and in some cases am seeing extremely slow writes (10s) for small amounts of data. I'm using memcache-client from rails on the client side. Has anyone seen this type of behavior from memcached before?

Answer 1

Not on any hardware I've ran, however I've never used it in a virtualized environment. I'd recommend checking out the pings between the servers assuming they're on separate instances. I'd also try to make sure you're not swapping or otherwise limited on memory.

Answer 2

That memcached client is known to be unnecessarily incredibly slow.

Perhaps trying Evan Weaver's client would give you better results.

Answer 3

I'm using EC2 at the moment and experiencing similar problems with the memcache. Sometimes the memcache responds in 50-100ms - other times it's over 10sec!

Doing a ping between the machines doesn't show any reason for the latency. I'd be interested to hear how you resolved the problem?

Answer 4

What Rails version are you using? The memcache-client that came with Rails 2.1.2 is version 1.5. There are a number of issues that have been resolved with the latest version, 1.7.x.

If you're using frozen Rails, just install the latest memcache-client gem and remove the memcache-client in RAILS_ROOT/vendor/rails/activesupport/lib/active_support/vendor/

Also, are you using cache_fu? I find that cache_fu issues a lot of unnecessary memcache calls. This is especially a problem if you have pages that show multiple records (e.g. search results). My solution to this was to stop using cache_fu and use plain Rails.cache.

Answer 5

Which version of memcached are you using? There are many bugs in the 1.2 series fixed in the 1.4 series. Did you configure memcached to use more memory than the EC2 VM has availble? Memcached is kind of pointless when you start swapping.

Instead of doing a "roll your own" memcached EC2, you could try using the Gear6 memcached AMI. It is free (except for the Amazon charges) for the 32bit instance types, and it gives you a nice managment and monitoring interface, so you can get better visibility into these kinds of problems.