What can cause a two minute delay in calling a webservice?

Question

We have a .NET 3.5 app that calls a webservice on server. In nearly every installation of this app, the whole request/reply process takes about half a second.

In one particular installation, these requests are mysteriously taking almost exactly 85 seconds (within half a second).

My first thought was that the webservice client was rebuilding the XML serialization assembly each call, but even sending a hardcoded xml file directly still takes almost exactly the same amout of time. Watching the network traffic seems to indicate that the actual sending data part of the transaction is happening in under a second. So the problem is all on the client side.

Is there some sort of permissions delay that could be causing this issue?

Edit (more detail):
The application is a basic wrapper around the webservice queries - type in some parameters, send a query to the webservice and get a response. We started with client code generated by the wsdl.exe tool, but also tried just using HttpWebRequest directly when we encountered the issues. From following the logs and the network traces, the flow seems to be:

T 0:00 - user initiates request
T 1:24 - the application sends request to server
T 1:25 - the client receives the response and displays to the user.

Answer 1

If the data is sent in under a second but the total response time is 85 seconds, why do you feel the problem is on the client side? Does it take 85 seconds to receive the reply from the server?

Generally, if you see that type of very consistent (within half a second) delay, look for some sort of timeout in your processing. This sounds very much like a network timeout. Is there a network resource that the server might be trying to access? Authentication is one such network resource. A file share, external http or ftp call could also be candidates.

Can you describe your application in more detail?

Answer 2

It does sound like a timeout issue. One thought- what type of proxy is configured? Is the machine trying to detect an HTTP proxy before it does the request? I'd think its time to use a network monitor to see whats happening.

Answer 3

If the service is also .NET you might be able to pinpoint the source of the problem using tracing. You can add traces to your web.config as follows:

 <system.diagnostics>
        <sources>
            <source name="System.ServiceModel"
                    switchValue="Information, ActivityTracing"
                    propagateActivity="true">
                <listeners>
                    <add name="traceListener"
                        type="System.Diagnostics.TextWriterTraceListener"
                        initializeData="c:\temp\Traces.txt"  />
                </listeners>
            </source>
        </sources>
    </system.diagnostics>

This will trace any activity of 'System.ServiceModel' which is webservices in 3.5

Answer 4

Use a program like wireshark to sniff the network when doing the request. First at the client and if you're still in the dark at the server. (both at the same time gives even more insight).

This way you can rule out, tcp-ip retransmissions, authentication issues, proxy weirdness, etc etc

Answer 5

Thanks for all your suggestions. The problem was being caused by proxy auto-detection. Basically, if internet explorer is set to auto-detect proxies, then HttpWebRequest will as well, only it will auto-detect every time you create a new WebRequest instead of caching anything in a helpful manner.

Eventually, I found this KB article: http://support.microsoft.com/kb/968699

The solution was simply to add this to the app.config file:

<configuration>
    <system.net>
        <defaultProxy >
            <!-- Disable Autoproxy-->
            <proxy autoDetect="false"/>
        </defaultProxy>
    </system.net>
</configuration>