Asynchronous WCF calls timeout on client side after 10 seconds

Question

EDIT

After looking at this for a while, I thought it might be a configuration issue on my development box. However, after doing a clean download of the source code to a different development machine, I'm still getting this issue.

I've got a Silverlight client that calls WCF services asynchronously. Intermittently, I will get one of the generic NotFound exceptions. The exceptions (which notoriously lack detail) happen intermittently on pretty much any of the services that I call.

Here's the thing. With judicious setting of breakpoints, I've been able to determine that the service-side is executing normally. Data is being retrieved and returned. The problem seems to be more on the client side of things.

Here's the rub... I can consistently make the exception happen if I make the service execute for more than 10 seconds. When I do, it never makes it back to my completed callback. Instead, I get the exception in the client-side Reference.cs for the service:

        public System.Collections.ObjectModel.ObservableCollection<Project.Ui.SilverLight.ServiceName.ModelName> EndGetService(System.IAsyncResult result) {
            object[] _args = new object[0];
            System.Collections.ObjectModel.ObservableCollection<roject.Ui.SilverLight.ServiceName.ModelName> _result = ((System.Collections.ObjectModel.ObservableCollection<roject.Ui.SilverLight.ServiceName.ModelName>)(base.EndInvoke("GetService", _args, result)));
            return _result;
        }

The exception I get is (not very helpful):

System.ServiceModel.CommunicationException was unhandled by user code
  Message=The remote server returned an error: NotFound.
  StackTrace:
       at System.ServiceModel.AsyncResult.End[TAsyncResult](IAsyncResult result)
       at System.ServiceModel.Channels.ServiceChannel.EndCall(String action, Object[] outs, IAsyncResult result)
       at System.ServiceModel.ClientBase`1.ChannelBase`1.EndInvoke(String methodName, Object[] args, IAsyncResult result)
       at Project.Ui.SilverLight.Service.ServicesClient.ServicesClientChannel.EndGetxxxxx(IAsyncResult result)
       at Project.Ui.SilverLight.Service.ServicesClient.Project.Ui.SilverLight.Service.IServices.EndGetxxxx(IAsyncResult result)
       at Project.Ui.SilverLight.Service.ServicesClient.OnEndGet(IAsyncResult result)
       at System.ServiceModel.ClientBase`1.OnAsyncCallCompleted(IAsyncResult result)
  InnerException: System.Net.WebException
       Message=The remote server returned an error: NotFound.
       StackTrace:
            at System.Net.Browser.AsyncHelper.BeginOnUI(SendOrPostCallback beginMethod, Object state)
            at System.Net.Browser.BrowserHttpWebRequest.EndGetResponse(IAsyncResult asyncResult)
            at System.ServiceModel.Channels.HttpChannelFactory.HttpRequestChannel.HttpChannelAsyncRequest.CompleteGetResponse(IAsyncResult result)
       InnerException: System.Net.WebException
            Message=The remote server returned an error: NotFound.
            StackTrace:
                 at System.Net.Browser.BrowserHttpWebRequest.InternalEndGetResponse(IAsyncResult asyncResult)
                 at System.Net.Browser.BrowserHttpWebRequest.<>c__DisplayClass5.<EndGetResponse>b__4(Object sendState)
                 at System.Net.Browser.AsyncHelper.<>c__DisplayClass2.<BeginOnUI>b__0(Object sendState)
            InnerException: 

Binding info (names changed, but they do match the services being executed)

    <binding name="Project.WebUI.Services.xxxxxServices.customBinding0" closeTimeout="00:01:00"
      openTimeout="00:01:00" receiveTimeout="00:10:00" sendTimeout="00:01:00">
      <binaryMessageEncoding/>
      <httpTransport />
    </binding>

...

  <service name="Project.WebUI.Services.xxxxxServices">
    <endpoint address="" binding="customBinding" bindingConfiguration="Project.WebUI.Services.xxxxxServices.customBinding0"
      contract="Project.WebUI.Services.xxxxxServices" />
    <endpoint address="mex" binding="mexHttpBinding" contract="IMetadataExchange" />
  </service>

I believe I've checked the relevant timeouts. The channel's operationTimeout is set to at least one minute, as is the ReceiveTimeout and OpenTimeout. Is there something arcane about Silverlight async WCF calls that needs to be set to tell it to go more than ten seconds?

Answer 1

In the config files above, it looks like they're from the server's configuration. Have you checked the client-side configuration as well? The SendTimeout and ReceiveTimeout properties on the client channel both default to one minute, but it's possible they could have been adjusted or overridden. And of course, you should check both your configuration XML files, but also where you create the binding in code, as it's possible to set properties there which conflict with your configuration files. (On both the client and on the server.)

Answer 2

Could it be something on the network.

In this they got the same error message but it was really a 504 Gateway timeout.

If for example you have a proxy server between the silverlight app and the server, that closes the connection so that the callback cannot reach the client.

Answer 3

Just out of curiosity, what is the idleTimeout in web.config set at the moment? Is it possibly set to 10 minutes??

UPDATE

So here is your problem! This is a known issue with idleTimeout! Just remove that setting and things should work again.

Answer 4

The reason why you are gettign not found is NOT due to a time out. I lost a week trying to figure this out.

I have references to silverlight but this is not a silverlight specific problem.

The not found fault exists because of the way the error codes are produced. You need to intercept any faultsand change the error code - this will cause the errors to be reported properly and not fuiltered out due to security.

If you think about it, the server is not found because the call is asynchronous and it has died due to an exception in your code.

I use this class in my wcf service to plug in - i am doing this through the web.config, however its not hard to change this so that its done wit ha class level attribute on the service itself.

you will probably want to strip out all the Contract stuff. Its being used in the project i am working on but is not related to this problem in any way.

 [AttributeUsage(AttributeTargets.Class)]
    public class FaultBehavior : Attribute, IServiceBehavior,  IEndpointBehavior
    {  

    public class SilverlightFaultMessageInspector : IDispatchMessageInspector
    {

        public void BeforeSendReply(ref Message reply, object correlationState)
        {
            Contract.Assume(reply !=null );
            if (reply.IsFault)
            {
                HttpResponseMessageProperty property = new HttpResponseMessageProperty();

                // Here the response code is changed to 200.
                property.StatusCode = System.Net.HttpStatusCode.OK;

                Contract.Assume(reply.Properties != null);
                reply.Properties[HttpResponseMessageProperty.Name] = property;
            }
        }

        public object AfterReceiveRequest(ref Message request, IClientChannel channel, InstanceContext instanceContext)
        {
            // Do nothing to the incoming message.
            return null;
        }


    }


    #region IServiceBehavior Members

    void IServiceBehavior.AddBindingParameters(ServiceDescription serviceDescription, ServiceHostBase serviceHostBase, Collection<ServiceEndpoint> endpoints, BindingParameterCollection bindingParameters)
    {

    }

    void IServiceBehavior.ApplyDispatchBehavior(ServiceDescription serviceDescription, ServiceHostBase serviceHostBase)
    {
        Contract.Assume( serviceHostBase != null);
        Contract.Assume(serviceHostBase.ChannelDispatchers != null);

        foreach (ChannelDispatcher cDispatcher in serviceHostBase.ChannelDispatchers)
        {
            Contract.Assume(cDispatcher != null);
            Contract.Assume(cDispatcher.Endpoints != null );
            foreach (EndpointDispatcher endpointDisbatcher in cDispatcher.Endpoints)
            {
                Contract.Assume(endpointDisbatcher != null);
                Contract.Assume(endpointDisbatcher.DispatchRuntime  != null);
                Contract.Assume(endpointDisbatcher.DispatchRuntime.MessageInspectors  != null);
                endpointDisbatcher.DispatchRuntime.MessageInspectors.Add(new SilverlightFaultMessageInspector());
            }
        }
    }

    void IServiceBehavior.Validate(ServiceDescription serviceDescription, ServiceHostBase serviceHostBase)
    {

    }

    #endregion

    #region IEndpointBehavior Members

    void IEndpointBehavior.AddBindingParameters(ServiceEndpoint endpoint, BindingParameterCollection bindingParameters)
    {

    }

    void IEndpointBehavior.ApplyClientBehavior(ServiceEndpoint endpoint, ClientRuntime clientRuntime)
    {

    }

    void IEndpointBehavior.ApplyDispatchBehavior(ServiceEndpoint endpoint, EndpointDispatcher endpointDispatcher)
    {
        Contract.Assume(endpointDispatcher != null);
        Contract.Assume(endpointDispatcher.DispatchRuntime != null);
        Contract.Assume(endpointDispatcher.DispatchRuntime.MessageInspectors != null);
        SilverlightFaultMessageInspector inspector = new SilverlightFaultMessageInspector();
        endpointDispatcher.DispatchRuntime.MessageInspectors.Add(inspector);
    }

    void IEndpointBehavior.Validate(ServiceEndpoint endpoint)
    {

    }

    #endregion
}

you then need to define the element that will appear in the web.config liek this

public class FaultHandlerElement   : BehaviorExtensionElement 
{
    protected override object CreateBehavior()
    {
        return new FaultBehavior();
    }

    public override Type BehaviorType
    {
        get { return typeof(FaultBehavior); }
    }


}

and then in the web config you need to add this to the service model section - fault exception will have a wavy line under it ;)

<system.serviceModel>
    <extensions>
        <behaviorExtensions>
            <add name="faultBehaviourExtension"
  type="CopSilverlight.Web.FaultHandlerElement, CopSilverlight.Web, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null"/>
        </behaviorExtensions>
    </extensions

you then hook your behaviours up to use this like so

   <serviceBehaviors>
                <behavior name="basicHttpBehaviour">
                    <serviceMetadata httpGetEnabled="true" />
                    <serviceDebug includeExceptionDetailInFaults="true" />
                    <faultBehaviourExtension />
                </behavior>

An alternative 'fix' is to turn on tracing in your wcf services oen thing to note is that if you do not make the directory it will simply not produce a log

    <system.diagnostics>
        <sources>
            <source name="System.ServiceModel"
                switchValue="Information, ActivityTracing"
                propagateActivity="true">
                <listeners>
                    <add name="traceListener"
  type="System.Diagnostics.XmlWriterTraceListener"
  initializeData="c:\logs\appName\wcf.svclog"  />
                </listeners>
            </source>
        </sources>
    </system.diagnostics>

the logging is no way ideal though - the first solution actually fixes the problem.

I have implemented both of these and now i get exceptions bubbling up to my silverlight app as you would expect.