C# - Locking issues with Mutex

Question

I've got a web application that controls which web applications get served traffic from our load balancer. The web application runs on each individual server.

It keeps track of the "in or out" state for each application in an object in the ASP.NET application state, and the object is serialized to a file on the disk whenever the state is changed. The state is deserialized from the file when the web application starts.

While the site itself only gets a couple requests a second tops, and the file it rarely accessed, I've found that it was extremely easy for some reason to get collisions while attempting to read from or write to the file. This mechanism needs to be extremely reliable, because we have an automated system that regularly does rolling deployments to the server.

Before anyone makes any comments questioning the prudence of any of the above, allow me to simply say that explaining the reasoning behind it would make this post much longer than it already is, so I'd like to avoid moving mountains.

That said, the code that I use to control access to the file looks like this:

    internal static Mutex _lock = null;
    /// <summary>Executes the specified <see cref="Func{FileStream, Object}" /> delegate on the filesystem copy of the <see cref="ServerState" />.
    /// The work done on the file is wrapped in a lock statement to ensure there are no locking collisions caused by attempting to save and load
    /// the file simultaneously from separate requests.
    /// </summary>
    /// <param name="action">The logic to be executed on the <see cref="ServerState" /> file.</param>
    /// <returns>An object containing any result data returned by <param name="func" />.</returns>
    private static Boolean InvokeOnFile(Func<FileStream, Object> func, out Object result)
    {
        var l = new Logger();
        if (ServerState._lock.WaitOne(1500, false))
        {
            l.LogInformation("Got lock to read/write file-based server state.", (Int32)VipEvent.GotStateLock);
            var fileStream = File.Open(ServerState.PATH, FileMode.OpenOrCreate, FileAccess.ReadWrite, FileShare.None);                
            result = func.Invoke(fileStream);                
            fileStream.Close();
            fileStream.Dispose();
            fileStream = null;
            ServerState._lock.ReleaseMutex();
            l.LogInformation("Released state file lock.", (Int32)VipEvent.ReleasedStateLock);
            return true;
        }
        else
        {
            l.LogWarning("Could not get a lock to access the file-based server state.", (Int32)VipEvent.CouldNotGetStateLock);
            result = null;
            return false;
        }
    }

This usually works, but occasionally I cannot get access to the mutex (I see the "Could not get a lock" event in the log). I cannot reproduce this locally - it only happens on my production servers (Win Server 2k3/IIS 6). If I remove the timeout, the application hangs indefinitely (race condition??), including on subsequent requests.

When I do get the errors, looking at the event log tells me that the mutex lock was achieved and released by the previous request before the error was logged.

The mutex is instantiated in the Application_Start event. I get the same results when it is instantiated statically in the declaration.

Excuses, excuses: threading/locking is not my forté, as I generally don't have to worry about it.

Any suggestions as to why it randomly would fail to get a signal?

---

Update:

I've added proper error handling (how embarrassing!), but I am still getting the same errors - and for the record, unhandled exceptions were never the problem.

Only one process would ever be accessing the file - I don't use a web garden for this application's web pool, and no other applications use the file. The only exception I can think of would be when the app pool recycles, and the old WP is still open when the new one is created - but I can tell from watching the task manager that the issue occurs while there is only one worker process.

@mmr: How is using Monitor any different from using a Mutex? Based on the MSDN documentation, it looks like it is effectively doing the same thing - if and I can't get the lock with my Mutex, it does fail gracefully by just returning false.

Another thing to note: The issues I'm having seem to be completely random - if it fails on one request, it might work fine on the next. There doesn't seem to be a pattern, either (certainly no every other, at least).

---

Update 2:

This lock is not used for any other call. The only time _lock is referenced outside the InvokeOnFile method is when it is instantiated.

The Func that is invoked is either reading from the file and deserializing into an object, or serializing an object and writing it to the file. Neither operation is done on a separate thread.

ServerState.PATH is a static readonly field, which I don't expect would cause any concurrency problems.

I'd also like to re-iterate my earlier point that I cannot reproduce this locally (in Cassini).

---

Lessons learned:

• Use proper error handling (duh!)
• Use the right tool for the job (and have a basic understanding of what/how that tool does). As sambo points out, using a Mutex apparently has a lot of overhead, which was causing issues in my application, whereas Monitor is designed specifically for .NET.

Answer 1

You should only be using Mutexes if you need cross-process synchronization.

Although a mutex can be used for intra-process thread synchronization, using Monitor is generally preferred, because monitors were designed specifically for the .NET Framework and therefore make better use of resources. In contrast, the Mutex class is a wrapper to a Win32 construct. While it is more powerful than a monitor, a mutex requires interop transitions that are more computationally expensive than those required by the Monitor class.

If you need to support inter-process locking you need a Global mutex.

The pattern being used is incredibly fragile, there is no exception handling and you are not ensuring that your Mutex is released. That is really risky code and most likely the reason you see these hangs when there is no timeout.

Also, if your file operation ever takes longer than 1.5 seconds then there is a chance concurrent Mutexes will not be able to grab it. I would recommend getting the locking right and avoiding the timeout.

I think its best to re-write this to use a lock. Also, it looks like you are calling out to another method, if this take forever, the lock will be held forever. That's pretty risky.

This is both shorter and much safer:

        // if you want timeout support use 
        // try{var success=Monitor.TryEnter(m_syncObj, 2000);}
        // finally{Monitor.Exit(m_syncObj)}
        lock(m_syncObj)
        {
            l.LogInformation("Got lock to read/write file-based server state.", (Int32)VipEvent.GotStateLock);
            using (var fileStream = File.Open(ServerState.PATH, FileMode.OpenOrCreate, FileAccess.ReadWrite, FileShare.None))
            {
                // the line below is risky, what will happen if the call to invoke
                // never returns? 
                result = func.Invoke(fileStream);
            }
        }

        l.LogInformation("Released state file lock.", (Int32)VipEvent.ReleasedStateLock);
        return true;

        // note exceptions may leak out of this method. either handle them here.
        // or in the calling method. 
        // For example the file access may fail of func.Invoke may fail

Answer 2

If some of the file operations fail, the lock will not be released. Most probably that is the case. Put the file operations in try/catch block, and release the lock in the finally block.

Anyway, if you read the file in your Global.asax Application_Start method, this will ensure that noone else is working on it (you said that the file is read on application start, right?). To avoid collisions on application pool restaring, etc., you just can try to read the file (assuming that the write operation takes an exclusive lock), and then wait 1 second and retry if exception is thrown.

Now, you have the problem of synchronizing the writes. Whatever method decides to change the file should take care to not invoke a write operation if another one is in progress with simple lock statement.

Answer 3

I see a couple of potential issues here.

Edit for Update 2: If the function is a simple serialize/deserialize combination, I'd separate the two out into two different functions, one into a 'serialize' function, and one into a 'deserialize' function. They really are two different tasks. You can then have different, lock-specific tasks. Invoke is nifty, but I've gotten into lots of trouble myself going for 'nifty' over 'working'.

1) Is your LogInformation function locking? Because you call it inside the mutex first, and then once you release the mutex. So if there's a lock to write to the log file/structure, then you can end up with your race condition there. To avoid that, put the log inside the lock.

2) Check out using the Monitor class, which I know works in C# and I'd assume works in ASP.NET. For that, you can just simply try to get the lock, and fail gracefully otherwise. One way to use this is to just keep trying to get the lock. (Edit for why: see here; basically, a mutex is across processes, the Monitor is in just one process, but was designed for .NET and so is preferred. No other real explanation is given by the docs.)

3) What happens if the filestream opening fails, because someone else has the lock? That would throw an exception, and that could cause this code to behave badly (ie, the lock is still held by the thread that has the exception, and another thread can get at it).

4) What about the func itself? Does that start another thread, or is it entirely within the one thread? What about accessing ServerState.PATH?

5) What other functions can access ServerState._lock? I prefer to have each function that requires a lock get its own lock, to avoid race/deadlock conditions. If you have many many threads, and each of them try to lock on the same object but for totally different tasks, then you could end up with deadlocks and races without any really easily understandable reason. I've changed to code to reflect that idea, rather than using some global lock. (I realize other people suggest a global lock; I really don't like that idea, because of the possibility of other things grabbing it for some task that is not this task).

    Object MyLock = new Object();
    private static Boolean InvokeOnFile(Func<FileStream, Object> func, out Object result)
{
    var l = null;
    var filestream = null;
    Boolean success = false;
    if (Monitor.TryEnter(MyLock, 1500))
        try {
            l = new Logger();
            l.LogInformation("Got lock to read/write file-based server state.", (Int32)VipEvent.GotStateLock);
            using (fileStream = File.Open(ServerState.PATH, FileMode.OpenOrCreate, FileAccess.ReadWrite, FileShare.None)){                
                result = func.Invoke(fileStream); 
            }    //'using' means avoiding the dispose/close requirements
            success = true;
         }
         catch {//your filestream access failed

            l.LogInformation("File access failed.", (Int32)VipEvent.ReleasedStateLock);
         } finally {
            l.LogInformation("About to released state file lock.", (Int32)VipEvent.ReleasedStateLock);
            Monitor.Exit(MyLock);//gets you out of the lock you've got
        }
    } else {
         result = null;
         //l.LogWarning("Could not get a lock to access the file-based server state.", (Int32)VipEvent.CouldNotGetStateLock);//if the lock doesn't show in the log, then it wasn't gotten; again, if your logger is locking, then you could have some issues here
    }
  return Success;
}