Drupal site - Memcache Connection errors

Question

We are tying to perf tune our drupal site. We are using Siege to measure performance (as drupal visitor).

Env: Nginx + FastCGI+ Memcache

Siege runs fine for a few seconds, and then we run into connection errors: Example:

HTTP/1.1 200  29.18 secs:    5877 bytes ==> /
HTTP/1.1 200  29.39 secs:    5877 bytes ==> /
warning: socket: -1656235120 select timed out: Connection timed out
warning: socket: -1673020528 select timed out: Connection timed out

Using the same Siege test confiuration, Nginx + FastCGI+ Drupal Cache seems to work fine. Example:

HTTP/1.1 200   1.41 secs:    5868 bytes ==> /
HTTP/1.1 200   1.40 secs:    5868 bytes ==> /

As you can see, Response time is much higher with MemCache, in addition to the connection errors.

Any idea what could be wrong here... and why Drupal is throwing errors with memcache under load?

Memcache runs on a separate instance. Allocated 2GB memory for MemCache.

Answer 1

I guess that You run out of memcached connections. Please run a check of Your memcached installation with a simple script every second. Then start Siege. I guess Your memcached stops responding after a while.

Test memcache php script:

<?php
 $memcache = new Memcache;
 $memcache->connect('localhost', 11211) or die ('Unable to connect');
 $version = $memcache->getVersion();
 echo 'Server version: '.$version;
?>

What I guess is happening is that You have not disable the persistent connections in memcache and they hang around in the php threads. Memcached can serve ~1023 of them at a time and that might not be enough while Sieging.

You might also try ab, apache benchmarking tool with the close look to the -c switch. Play around with it and see how the results change on different values.

Finally, You should run a tcpdump on Your memcached port (usually 11211) on the php machine to find out what is happening to the connections. Does drupal start them? Does the other host respond with a RST or does it time out?

There was a bug in the memcached php documentation api that said that the connections are non-persistent by default. They are persistent by default (well, they were at the time I had the problem with it).

Feel free to comment this answer, I'll read the comments and assist further if necessary.