.Net Intermittent System.Web.Services.Protocols.SoapHeaderException

Question

We have a .net 3.5 web app that consumes third party web services. The proxy was created by adding a web reference to their wsdl. This proxy is not compiled.

Our error logging is picking up frequent but intermittent exceptions:

An exception of type 'System.Web.Services.Protocols.SoapHeaderException' occurred and was caught

If I follow the url to the page that generated the exception, I can't recreate it.

Edit: Here is most of the exception - where it bubbled up from

Message : Internal Error
Type : System.Web.Services.Protocols.SoapHeaderException, System.Web.Services, Version=2.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a Source : System.Web.Services Help link : 
Actor : 
Code : http://schemas.xmlsoap.org/soap/envelope/:Client
Detail : 
Lang : 
Node : 
Role : 
SubCode : 
Data : System.Collections.ListDictionaryInternal
TargetSite : System.Object[] ReadResponse(System.Web.Services.Protocols.SoapClientMessage, System.Net.WebResponse, System.IO.Stream, Boolean)
Stack Trace :    at System.Web.Services.Protocols.SoapHttpClientProtocol.ReadResponse(SoapClientMessage message, WebResponse response, Stream responseStream, Boolean asyncCall)
   at System.Web.Services.Protocols.SoapHttpClientProtocol.Invoke(String methodName, Object[] parameters)
   at Vendor.getSearch(getSearchRequest getSearchRequest) in c:\WINDOWS\Microsoft.NET\Framework\v2.0.50727\Temporary ASP.NET Files\root\be43c34e\b09edc7e\App_WebReferences.pww-cf-q.0.cs:line 73

Edit 2: Inner exceptions:

I sometimes get the following inner exceptions logged:

Message : Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host.
    Type : System.IO.IOException, mscorlib, Version=2.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089
    Source : System
    Help link : 
    Data : System.Collections.ListDictionaryInternal
    TargetSite : Int32 Read(Byte[], Int32, Int32)
    Stack Trace :    at System.Net.Sockets.NetworkStream.Read(Byte[] buffer, Int32 offset, Int32 size)
       at System.Net.FixedSizeReader.ReadPacket(Byte[] buffer, Int32 offset, Int32 count)
       at System.Net.Security.SslState.StartReceiveBlob(Byte[] buffer, AsyncProtocolRequest asyncRequest)
       at System.Net.Security.SslState.CheckCompletionBeforeNextReceive(ProtocolToken message, AsyncProtocolRequest asyncRequest)
       at System.Net.Security.SslState.StartSendBlob(Byte[] incoming, Int32 count, AsyncProtocolRequest asyncRequest)
       at System.Net.Security.SslState.ForceAuthentication(Boolean receiveFirst, Byte[] buffer, AsyncProtocolRequest asyncRequest)
       at System.Net.Security.SslState.ProcessAuthentication(LazyAsyncResult lazyResult)
       at System.Net.TlsStream.CallProcessAuthentication(Object state)
       at System.Threading.ExecutionContext.runTryCode(Object userData)
       at System.Runtime.CompilerServices.RuntimeHelpers.ExecuteCodeWithGuaranteedCleanup(TryCode code, CleanupCode backoutCode, Object userData)
       at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
       at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)
       at System.Net.TlsStream.ProcessAuthentication(LazyAsyncResult result)
       at System.Net.TlsStream.Write(Byte[] buffer, Int32 offset, Int32 size)
       at System.Net.PooledStream.Write(Byte[] buffer, Int32 offset, Int32 size)
       at System.Net.ConnectStream.WriteHeaders(Boolean async)

And/Or:

Message : An existing connection was forcibly closed by the remote host
        Type : System.Net.Sockets.SocketException, System, Version=2.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089
        Source : System
        Help link : 
        ErrorCode : 10054
        SocketErrorCode : ConnectionReset
        NativeErrorCode : 10054
        Data : System.Collections.ListDictionaryInternal
        TargetSite : Int32 Receive(Byte[], Int32, Int32, System.Net.Sockets.SocketFlags)
        Stack Trace :    at System.Net.Sockets.Socket.Receive(Byte[] buffer, Int32 offset, Int32 size, SocketFlags socketFlags)
           at System.Net.Sockets.NetworkStream.Read(Byte[] buffer, Int32 offset, Int32 size)

Update

We're still working on it. Originally there was a route issue, which was resolved. We're still getting the inner exception with socket errors. We had MS support involved today, and they looked at some traces and network captures. The web service host does round-robin DNS, and they may be responding on a different IP address for the syn syn/ack from one ip, and the next from a different ip. This is not good. This is likely quite specific to our situation, but perhaps it applies to others as well.

Microsoft Network Monitor and an application trace got us the information we needed.

Answer 1

This exception comes from the server side through SOAP headers and your client is probably not at fault here. You might get more info about this error from those 3rd party web service providers.

Answer 2

Assuming that the server is working properly, the fact that it included the Client code in the fault indicates that the server believes the client is at fault.

Are you passing SOAP Headers to the service? Then look to see if it's possible you're doing it incorrectly? Perhaps you're meant to be passing headers but sometimes you don't?

Answer 3

The exceptions you are seeing might be caused by errors in your server side code.

It looks like the exception occurred in Vendor.getSearch. What code does this method contain?

Answer 4

We recently ran into this with our own web services and it ended up being a timeout issue. The actual timeout of the web service was set high enough and never getting hit, but it was throwing this exception because it was not giving any updates to stay alive (non-pumping). This was the solution for us, hope it helps. You can subclasss your web service or put this straight into the reference.cs file, but it gets overwritten when you update web references.

public partial class TheWebServiceSubClass : TheWebService
{
    protected override WebRequest GetWebRequest(Uri uri)
    {
        HttpWebRequest webRequest = (HttpWebRequest) base.GetWebRequest(uri);

        webRequest.KeepAlive = false;
        webRequest.ProtocolVersion=HttpVersion.Version10;
        return webRequest;
    }
}

Answer 5

The reason for this is is that while reading the beginning of the response from the server (the SOAP headers that are at the start of the response), the server instead of sending a SOAP response, closed the TCP/IP connection.

Obviously the error is not on the client side. It is most likely a server overload situation where instead of serving a response or a 500 or 503 HTTP error code it simply terminates the TCP/IP connection. There is a (very small) probability that some network equipment is causing these connection drops, but I would naturally start investigating the server side first.

I recommend using a tool like Fiddler or Wireshark to log one of these errors. Since it's a forced TCP/IP connection close, I would look for communications with the server that end with TCP RST packet that forces the connection to close.