Memcache service is down - how to handle?

Question

I am using Memcache in my ASP.NET application. I have 3 memcache servers that it connects to. If all of them are down, there is a significant delay in loading the pages. When all servers are down, I would want to not use memcache.

When I was stepping through the code, I noticed that when the servers are down, memcache code does not throw an exception (I thought I could "Catch" it in my application and take the necessary steps).

How would I handle this situation?

Answer 1

How do one check if the service is down? There are several ways:

• Check if the process is running (if you are checking same machine). I doubt it is fast.
• Try to connect to socket and see if connection fails. I also doubt it is fast.

So, just take some measures to not let all of your memcached be down. Watch them with monit for example.

Answer 2

Since Enyim is Open Source, you could edit ServerPool.cs, like so:

    /// <summary>
    /// Marks a node as dead (unusable)
    ///  - moves hte node to the  "dead list"
    ///  - recreates the locator based on the new list of still functioning servers
    /// </summary>
    /// <param name="node"></param>
    private void MarkAsDead(MemcachedNode node)
    {
        this.serverAccessLock.UpgradeToWriterLock(Timeout.Infinite);

        try
        {
            // server gained AoeREZ while AFK?
            if (!node.IsAlive)
            {
                this.workingServers.Remove(node);
                this.deadServers.Add(node);

                // check if all servers are dead
                if (this.workingServers.Count == 0)
                    throw new NoMoreServersException();

                this.RebuildIndexes();
            }
        }
        finally
        {
            this.serverAccessLock.ReleaseLock();
        }
    }

I haven't tested above code. BTW, my changes are these three lines:

    // check if all servers are dead
    if (this.workingServers.Count == 0)
        throw new NoMoreServersException();

You should also create NoMoreServersException (or use a generic exception (not advisable though)).

Answer 3

What you are trying to do here is not really the job of your application.

Let’s assume your caching interface is something like this.

public class CacheManager
{
    protected ICacheProvider _repository;
    public CacheManager(ICacheProvider repository)
    {
        _repository = repository;
    }

    public void Store(string key, object data)
    {
        _repository.Store(key, data);
    }

    public void Destroy(string key)
    {
        _repository.Destroy(key);
    }

    public T Get<T>(string key)
    {
        return _repository.Get<T>(key);
    }
}

If this is the case, what are your choices if all memcache servers are down? You can only continue to live normal by accessing your db/filessystem/etc. After all, cache by definition is not reliable.

Assuming that your memcached is running on *nix servers, you should be looking at ways to monitor it from the network or OS level. Memcached has a wiki page for this. Write rules/scripts to parse and inform your admin and bring it up. Your simplest option is to use munin IMHO