WCF exception received on closing connection with callbacks in use

Question

Hey there all,

I am using a netNamedPipeBinding to perform inter-process WCF communication from a windows app to a windows service.

Now my app is running well in all other accounts (after fighting off my fair share of WCF exceptions as anybody who has worked with WCF would know..) but this error is one that is proving to be quite resilient.

To paint a picture of my scenario: my windows service could be queued to do some work at any given time through a button pressed in the windows app and it then talks over the netNamedPipeBinding which is a binding that supports callbacks (two-way communication) if you are not familiar and initiates a request to perform this work, (in this case a file upload procedure) it also throws the callbacks (events) every few seconds ranging from file progress to transfer speed etc. etc. back to the windows app, so there is some fairly tight client-server integration; this is how I receive my progress of what's running in my windows service back into my windows app.

Now, all is great, the WCF gods are relatively happy with me right now apart from one nasty exception which I receive every time I shutdown the app prematurely (which is a perfectly valid scenario). Whilst a transfer is in progress, and callbacks are firing pretty heavily, I receive this error:

System.ServiceModel.ProtocolException:
  The channel received an unexpected input message with Action 
  'http://tempuri.org/ITransferServiceContract/TransferSpeedChangedCallback' 
  while closing. You should only close your channel when you are not expecting 
  any more input messages.

Now I understand that error, but unfortunately I cannot guarantee to close my channel after never receiving any more input messsages, as the user may shutdown the app at any time therefore the work will still be continuing in the background of the windows service (kind of like how a virus scanner operates). The user should be able to start and close the win management tool app as much as they like with no interference.

Now the error, I receive immediately after performing my Unsubscribe() call which is the second last call before terminating the app and what I believe is the preferred way to disconnect a WCF client. All the unsubscribe does before closing the connection is simply removes the client id from an array which was stored locally on the win service wcf service (as this is an instance SHARED by both the win service and windows app as the win service can perform work at scheduled events by itself) and after the client id array removal I perform, what I hope (feel) should be a clean disconnection.

The result of this, besides receiving an exception, is my app hangs, the UI is in total lock up, progress bars and everything mid way, with all signs pointing to having a race condition or WCF deadlock [sigh], but I am pretty thread-savvy now and I think this is a relatively isolated situation and reading the exception as-is, I don't think it's a 'thread' issue per-se, as it states more an issue of early disconnection which then spirals all my threads into mayhem, perhaps causing the lock up.

My Unsubscribe() approach on the client looks like this:

    public void Unsubscribe()
    {
        try
        {
            // Close existing connections
            if (channel != null &&
                channel.State == CommunicationState.Opened)
            {
                proxy.Unsubscribe();
            }
        }
        catch (Exception)
        {
            // This is where we receive the 'System.ServiceModel.ProtocolException'.
        }
        finally
        {
            Dispose();
        }
    }

And my Dispose() method, which should perform the clean disconnect:

    public void Dispose()
    {
        // Dispose object
        if (channel != null)
        {
            try
            {
                // Close existing connections
                Close();
                // Attempt dispose object
                ((IDisposable)channel).Dispose();
            }
            catch (CommunicationException)
            {
                channel.Abort();
            }
            catch (TimeoutException)
            {
                channel.Abort();
            }
            catch (Exception)
            {
                channel.Abort();
                throw;
            }
        }
    }

And the WCF service Subscription() counterpart and class attributes (for reference) on the windows service server (nothing tricky here and my exception occurs client side):

    [ServiceBehavior(InstanceContextMode = InstanceContextMode.Single, 
    ConcurrencyMode = ConcurrencyMode.Multiple)] 
    public class TransferService : LoggableBase, ITransferServiceContract
    {
        public void Unsubscribe()
        {
            if (clients.ContainsKey(clientName))
            {
                lock (syncObj)
                {
                    clients.Remove(clientName);
                }
            }

#if DEBUG
            Console.WriteLine(" + {0} disconnected.", clientName);
#endif
        }
        ...
    }

Interface of:

[ServiceContract(
    CallbackContract = typeof(ITransferServiceCallbackContract), 
    SessionMode = SessionMode.Required)]
public interface ITransferServiceContract
{
    [OperationContract(IsInitiating = true)]
    bool Subscribe();

    [OperationContract(IsOneWay = true)]
    void Unsubscribe();
    ...
}

Interface of callback contract, it doesn't do anything very exciting, just calls events via delegates etc. The reason I included this is to show you my attributes. I did alleviate one set of deadlocks already by including UseSynchronizationContext = false:

[CallbackBehavior(UseSynchronizationContext = false, 
ConcurrencyMode = ConcurrencyMode.Multiple)]
public class TransferServiceCallback : ITransferServiceCallbackContract
{ ... }

Really hope somebody can help me! Thanks a lot =:)

Answer 1

OH my gosh, I found the issue.

That exception had nothing to do with the underyling app hang, that was just a precautionary exception which you can safely catch.

You would not believe it, I spent about 6 hours on and off on this bug, it turned out to be the channel.Close() locking up waiting for pending WCF requests to complete (which never would complete until the transfer has finished! which defeats the purpose!)

I just went brute-force breakpointing line after line, my issue was if I was too slow..... it would never hang, because somehow the channel would be available to close (even before the transfer had finished) so I had to breakpoint F5 and then quickly step to catch the hang, and that's the line it ended on. I now simply apply a timeout value to the Close() operation and catch it with a TimeoutException and then hard abort the channel if it cannot shut down in a timely fashion!

See the fix code:

private void Close()
{
    if (channel != null &&
        channel.State == CommunicationState.Opened)
    {
        // If cannot cleanly close down the app in 3 seconds,
        // channel is locked due to channel heavily in use
        // through callbacks or the like.
        // Throw TimeoutException
        channel.Close(new TimeSpan(0, 0, 0, 3));
    }
}

public void Dispose()
{
    // Dispose object
    if (channel != null)
    {
        try
        {
            // Close existing connections
            // *****************************
            // This is the close operation where we perform 
            //the channel close and timeout check and catch the exception.
            Close();

            // Attempt dispose object
            ((IDisposable)channel).Dispose();
        }
        catch (CommunicationException)
        {
            channel.Abort();
        }
        catch (TimeoutException)
        {
            channel.Abort();
        }
        catch (Exception)
        {
            channel.Abort();
            throw;
        }
    }
}

I am so happy to have this bug finally over and done with! My app is now shutting down cleanly after a 3 second timeout regardless of the current WCF service state, I hope I could have helped someone else who ever finds themselves suffering a similar issue.

Graham