The problem is in maximum number of concurrent connections to a single server in Internet Explorer 7/6. It is only 2! http://msdn.microsoft.com/en-us/library/cc304129(VS.85).aspx
If we have 3 (for ex.) concurrent service calls two of them will be sent to server immediately but the third will be waiting in queue. Also the send timer (corresponded to sendTimeout) is running when request is in queue. If the first two service requests will be running for a long time then the third will generate TimeoutException though it wasn't sent to the server (and we won't see any information about this request on the server side and cannot catch it with Fiddler...).
In more real situation if we have about 12 concurrent calls and default 1 min send timeout and if service calls process more than 10 seconds in average than we can easily get timeout exception with the last two calls (12 / 2 * 10 sec = 60 sec) because they will wait all others.
The solution is:
- Minimize number of concurrent service calls.
- Increase
sendTimeout value in client config.
- Implement auto retry feature for critical services.
- Implement queue of requests to manage them.
In my case I've done 1-3 things and that was enough.
Here is my implementation of auto retry feature:
public static class ClientBaseExtender
{
/// <summary>
/// Tries to execute async service call. If <see cref="TimeoutException"/> occured retries again.
/// </summary>
/// <typeparam name="TChannel">ServiceClient class.</typeparam>
/// <typeparam name="TArgs">Type of service client method return argument.</typeparam>
/// <param name="client">ServiceClient instance.</param>
/// <param name="tryExecute">Delegate that execute starting of service call.</param>
/// <param name="onCompletedSubcribe">Delegate that subcribes an event handler to the OnCompleted event of the service client method.</param>
/// <param name="onCompleted">Delegate that executes when service call is succeeded.</param>
/// <param name="onError">Delegate that executes when service call fails.</param>
/// <param name="maxAttempts">Maximum attempts to execute service call before error if <see cref="TimeoutException"/> occured (by default 5).</param>
public static void ExecuteAsyncRepeatedly<TChannel, TArgs>(this ClientBase<TChannel> client, Action tryExecute,
Action<EventHandler<TArgs>> onCompletedSubcribe, EventHandler<TArgs> onCompleted,
EventHandler<TArgs> onError, int maxAttempts)
where TChannel : class
where TArgs : AsyncCompletedEventArgs
{
int attempts = 0;
var serviceName = client.GetType().Name;
onCompletedSubcribe((s, e) =>
{
if (e.Error == null) // Everything is OK
{
if (onCompleted != null)
onCompleted(s, e);
((ICommunicationObject)client).Close();
Debug.WriteLine("[{1}] Service '{0}' closed.", serviceName, DateTime.Now);
}
else if (e.Error is TimeoutException)
{
attempts++;
if (attempts >= maxAttempts) // Final timeout after n attempts
{
Debug.WriteLine("[{2}], Final Timeout occured in '{0}' service after {1} attempts.", serviceName, attempts, DateTime.Now);
if (onError != null)
onError(s, e);
client.Abort();
Debug.WriteLine("[{1}] Service '{0}' aborted.", serviceName, DateTime.Now);
return;
}
// Local timeout
Debug.WriteLine("[{2}] Timeout occured in '{0}' service (attempt #{1}).", serviceName, attempts, DateTime.Now);
Debug.WriteLine("[{2}] Attempt #{0} to execute call to '{1}' service.", attempts + 1, serviceName, DateTime.Now);
tryExecute(); // Try again.
}
else
{
if (onError != null)
onError(s, e);
client.Abort();
Debug.WriteLine("[{1}] Service '{0}' aborted.", serviceName, DateTime.Now);
}
});
Debug.WriteLine("[{2}] Attempt #{0} to execute call to '{1}' service.", attempts + 1, serviceName, DateTime.Now);
tryExecute(); // First attempt to execute
}
}
And here is a usage:
var client = new MyServiceClient();
client.ExecuteAsyncRepeatedly(() => client.MyOperationAsync(...),
(EventHandler<MyOperationCompletedEventArgs> handler) => client.MyOperationCompleted += handler,
(s, e) => // OnCompleted
{
Do(e.Result);
},
(s, e) => // OnError
{
HandleError(e.Error);
}
);
Hope this will be helpful.