WCF : maxConcurrentCalls has exhausted

Question

Hello, I'm in the process of troubleshooting a WCF Service that hangs at some point. The service behavior is the following:

[ServiceBehavior( InstanceContextMode = InstanceContextMode.PerCall, ConcurrencyMode = ConcurrencyMode.Multiple )]

Throttling parameters are :

<serviceThrottling maxConcurrentCalls="50" maxConcurrentSessions="50" maxConcurrentInstances="50" />

Following is the service state taken from a hang dump :

0:000> !mdt 0000000000c9f270 -r
0000000000c9f270 (System.ServiceModel.Dispatcher.ServiceThrottle)
   calls:0000000000c9f3d8 (System.ServiceModel.Dispatcher.FlowThrottle)
      capacity:0x32 (System.Int32)
      count:0x32 (System.Int32)
      mutex:0000000000c9f418 (System.Object)
         <NO FIELDS>
      release:0000000000c9f398 (System.Threading.WaitCallback)
         _target:0000000000c9f270 (System.ServiceModel.Dispatcher.ServiceThrottle)
            <RECURSIVE>
         _methodBase:NULL (System.Reflection.MethodBase)
         _methodPtr:0000064273dddf30 (System.IntPtr)
         _methodPtrAux:0000000000000000 (System.IntPtr)
         _invocationList:NULL (System.Object)
         _invocationCount:0000000000000000 (System.IntPtr)
         <NO FIELDS>
      waiters:0000000000c9f430 (System.Collections.Generic.Queue`1[[System.Object, mscorlib]])
         _array:0000000028d73e70 (System.Object[], Elements: 16)
         _head:0x1 (System.Int32)
         _tail:0xA (System.Int32)
         _size:0x9 (System.Int32)
         _version:0x22 (System.Int32)
         _syncRoot:NULL (System.Object)
      propertyName:0000000000c9f2b8 (System.String: "MaxConcurrentCalls")
      configName:0000000000c9f358 (System.String: "maxConcurrentCalls")
   sessions:0000000000c9f508 (System.ServiceModel.Dispatcher.FlowThrottle)
      capacity:0x32 (System.Int32)
      count:0x9 (System.Int32)

....

   instanceContexts:000000000105ffc8 (System.ServiceModel.Dispatcher.FlowThrottle)
      capacity:0x32 (System.Int32)
      count:0x32 (System.Int32)

As you can see, maxConcurrentCalls has exhausted, while sessions count is only 9. I'm wondering if this could be caused by a malfunction in client code about proxy usage, like poor exception handling ?

Given a memory dump of the service, is there a way to find clients IP adresses ?

TIA.

Answer 1

This could be a combination of timeout and how you are using the wcf proxy / calling the wcf service.

Idea is that when you call the service, you do not tell the service to close, the connection will then hang around until it times-out after 10 mins. Therefore, you can make 10 calls per min that only take 1 sec, yet after 10 mins you have 100 concurrent connections.

Use of a "using" statement to create the proxy normally fixes this.

Answer 2

Found the source of the problem. MaxConcurrentCalls has exhausted because of a deadlock in the service code. We are using the c# lock keyword and it seems that sometimes the lock is not released when something weird happens in the code protected by the lock...

Anyway, thanks to everyone that has contributed to this thread.