Duplex WCF service hangs on 11th call

Question

I've got a duplex WCF service which hangs after the magic 10 proxy instantiations. The specific error on the client is:

"System.TimeoutException: This request operation sent to net.tcp://localhost:8080/RoomService/netTcp did not receive a reply within the configured timeout (00:00:59.9960000)".

There aren't any apparent error messages on the server.

Note that this isn't the standard, obvious problem, i.e., failing to close my proxy connections, as I'm closing every instance of my proxy connection appropriately before opening the next one:

try
{
    client.Close();
}
catch (CommunicationException)
{
    client.Abort();
}
catch (TimeoutException)
{
    client.Abort();
}
catch (Exception)
{
    client.Abort();
    throw;
}

And I've set my throttling behavior to 500 simultaneous everything:

ServiceThrottlingBehavior throttlingBehavior = new ServiceThrottlingBehavior()
{
    MaxConcurrentCalls = 500,
    MaxConcurrentSessions = 500,
    MaxConcurrentInstances = 500
};

I've set the ConcurrencyMode of my service to Multiple, and I've tried all three possible values for InstanceContextMode.

[ServiceBehavior(InstanceContextMode=InstanceContextMode.PerSession, ConcurrencyMode = ConcurrencyMode.Multiple)]

I've tried self-hosting the service, and hosting it within IIS, and I get the same results on each.

I've tried the NetTcpBinding, the WSDualHttpBinding, and the PollingDuplexBinding (on Silverlight), with the same results on each. I can't try the BasicHttpBinding or the WSHttpBinding, as this is a duplex service.

There was one place in my code where I was launching multiple threads (to execute multiple callbacks simultaneously), but for troubleshooting purposes I've commented that bit out, and it hasn't made a difference.

On the client, I've tried using new proxies for each test, and reusing the same proxy across all tests, but without any luck. I've tried creating a new InstanceContext for each proxy, and reusing the same InstanceContext across all proxies, and again, no luck.

Whatever I do, after the 10th test executed in my test harness, the next call to the service hangs.

Any thoughts on what I might be doing wrong?

Answer 1

I've had this happen when I had a semaphore on the server-side which wasn't being released after a client was done being serviced.

Are any server-side resources or locks not being released properly? Since your service instance is per session, I'd suspect that the server object is hanging around and holding a lock. What if you change the behavior to per call?

Answer 2

I don't have an answer but I do have some debugging advice.

1) Attach the visual studio debugger to the service process and see if it can catch whatever is going on.

2) Configure the service behavior to pass exception information back to the client and see if the service is throwing an exception that isn't being reported:

<behaviors>
    <serviceBehaviors>
        <behavior name="ServiceBehavior">
            <serviceDebug includeExceptionDetailInFaults="True"/>
        </behavior>
    </serviceBehaviors>
</behaviors>

3) Turn on service logging with ActivityTracing enabled and use the Service Trace Viewer (from Windows SDK) to analyze the log and see if anything pops up

<system.diagnostics>
 <trace autoflush="true" />
 <sources>
  <source name="System.ServiceModel" switchValue="Information, ActivityTracing" propagateActivity="true">
   <listeners>
    <add name="sdt" type="System.Diagnostics.XmlWriterTraceListener" initializeData="Service.svclog"  />
   </listeners>
  </source>
 </sources>
</system.diagnostics>

4) Isolate the service wrapper from the functional code and see if the service still hangs. If it doesn't then incrementally add functionality back until you figure out what is making it hang

5) If you are using an HTTP binding, proxy the service with fiddler and log the http traffic.

6) Try Hosting the WCF service in a Managed Windows Service and attach a debugger to the service process after you start it up.

Answer 3

OK, so I made at least one stupid mistake: I was creating the throttling behavior, but was neglecting to add it to the service proper. It's now added correctly:

ServiceThrottlingBehavior throttlingBehavior = new ServiceThrottlingBehavior()
{
    MaxConcurrentCalls = 500,
    MaxConcurrentSessions = 500,
    MaxConcurrentInstances = 500
};
base.Description.Behaviors.Add(throttlingBehavior);

And now I can run more than 10 tests, and hence my immediate problem is solved.

But I'm still puzzled as to why I'm running into this problem at all, since I'm specifically closing one proxy before moving on to the next. A MaxConcurrentXXX of 2 ought to be working in this scenario; I shouldn't need a MaxConcurrentXXX of 500. I'm a tad worried about scalability if every client that connects to the server continues to chew up a connection beyond the time that it's actually connected.

Maybe I'm making a stupid mistake somewhere else -- it wouldn't be the first time -- but I've specifically stepped through the code that's closing the proxy, and it's definitely getting called.

Answer 4

Take a look at ServiceBehavior.AutomaticSessionShutdown.

Answer 5

At the begining of the Callback function in the client use OperationContext.Current.Channel.Close();

This will solev the problem.