WebRequest and Resources - ASP.NET Timeout

Question

I have a ASP.NET Web Forms application that internally makes many SOAP and REST calls to web services. The SOAP calls are made using Microsoft's own "wrapping" code. The REST calls are made from a simple REST client. This client uses Using blocks to dispose of any resources.

The application runs fine for a few hours but then hangs up. The site will no longer be able to serve up asp.net pages. I tested the site by putting a hello.htm file on it... it served up fine. So it is definitely a problem in the ASP.NET engine space.

It feels like it has run out of webrequests from a pool. I could be way off. What can I try? What should I look at? It takes hours to get the problem to repro. Restarting the site on IIS fixes the issue, but of cource is not really a fix we can live with.

start of error and stack trace:
Server Error in '/' Application.

The operation has timed out Description: An unhandled exception occurred during the execution of the current web request. Please review the stack trace for more information about the error and where it originated in the code.

Exception Details: System.Net.WebException: The operation has timed out

Source Error: An unhandled exception was generated during the execution of the current web request. Information regarding the origin and location of the exception can be identified using the exception stack trace below.

Stack Trace: [WebException: The operation has timed out] System.Net.HttpWebRequest.GetRequestStream() +5322142 System.Web.Services.Protocols.SoapHttpClientProtocol.Invoke(String methodName, Object[] parameters) +103

UPDATE and MORE INFO
The homegrown REST calls and SOAP are all done synchronous.
They are not long-lived they take about 1 sec.
Web Services are hosted on a Tomcat server (different machine) in the same data center.
The .NET app calls SOAP and REST services on the Tomcat server.

The app somehow fixed itself after about an hour of staying in this hung state. Thoughts? How do I monitor the .NET Threadpool? Does DefaultConnectionLimit affect the self-initiated outbound connections?

RESOLUTION see additional comments to answers and... 1) close all streams and responses/requests from your homegrown REST code (oops)
2) System.Net.ServicePointManager.DefaultConnectionLimit = 96;
// default = 12 times # of cores
System.Net.ServicePointManager.MaxServicePointIdleTime= 3000;
// default = 100000 (100 seconds)
By fixing any leaky code and timing out the IdleTime faster everything seems to work fine now

Answer 1

1) Are you using asynchronous WebRequests? 2) Are your web service calls long lived (meaning they take a long time to complete?) 3) Where is the webservice hosted? Is it on the same machine? 4) Where is the REST service hosted? Same machine as asp.net app or different?

When a request comes into an ASP.NEt server, it is handled on a thread pool thread. I am not sure if this will be a completion port thread or a thread pool thread.

Anyway, your app is being invoked on a threadpool thread. On this thread you are making an outbound HTTPWebRequest (HWR). If this request is synchronous, it should not take any extra thread. However if it is asynchronous it will take another thread from the pool to complete.

Now, if the HWR you made is back to the same server, and that request needs another thread on the same server to complete, you now have a requirement for one more thread. Combine this with the Webservice taking a long time, and you are tying up atleast 2 (max 3) threads per request. If you are issuing a lot of request to the asp.net server, the server can quickly hit up against the max threadpool thread limit.

So far, I am assuming that there are no logic bugs in your code, and you are disposing of your HttpWebResponse objects correctly.

So, when there is no thread to finish the work, your HttpWebRequest can throw a timeout exception.

This might also happen if your backend webservice calls are to the same machine, and you have not increased the ServicePointManager.DefaultConnectionLimit to a reasonable value for your scenario. For eg, if your asp.net app expects to be hit with 200 simultaneous requests, you should set the DefaultConnectionLimit = 200 + some reasonable buffer. Lack of connection in the connectiion pool can also cause this.

Solution:

I would first up the DefaultConnectionLimit. If that doesnt solve it, then you would need to monitor ASP.NET performance counters and see whether the .Net threadpool is being exhausted which could contribute to webrequests failing.