Diagnosing "Request timed out" HttpExceptions

Question

Here on StackOverflow, we're seeing a few "Request timed out" exceptions every day.

The facts:

• Request timeout is the default 90 seconds
• Occurs only on POSTs
• Data posted is text, usually small (< 1KB), but can range to a few KB
• No Form data is captured in server variables
• Client UAs are diverse: IE5.5 - 7, Firefox 3.0.5, iPhone, Chrome
• Client locations are diverse: UK, France, USA - NC, OH, NE, IN

We've tested a server-based timeout (i.e. using Thread.Sleep) and all form variables are correctly captured in the exception log - this leads us to believe the client is having issues sending the request in the allotted time.

Any thoughts on how to trap/debug this condition are very welcome!

Answer 1

Have you tried posting manually via telnet and simply not completing the POST. I'd be interested to see if you could replicate the behavior you are seeing. Given the nature of the site, I wouldn't be surprised if you were getting a few malformed POSTs intentionally to try and hack the system.

I have noticed on occasion that I need to restart Safari in order to get SO working again after some action hangs, but I assumed it was my problem.

Answer 2

We used to see those a lot with our very high traffic web client -- wonder if it's related. What supposedly was happening was the HttpWebRequest (I'm assuming you're having problems with HttpWebResponse? Maybe they have the same issues) uses some janky thread pool underneath the covers, even when your requests are synchronous. Every now and then something would deadlock because some other .NET object higher in the stack was using the same system threadpool and one would starve the other out, eventually causing a timeout. I think the issue is better described here: http://www.deez.info/sengelha/2005/03/03/beware-threadpools-and-httpwebrequest/

Answer 3

If you are running IIS 7 you could use the Failed Request Tracing. I haven't actually used it for timeouts, I mostly have set it up to capture just specific http error codes. But I know you can get it to dump traces of any request taking more than X amount of time.

Answer 4

I would also scan the IP addresses in your logs to see if it's the same people having issues repeatedly. You know, it's just possible that some people are still using dial up accounts, or there may be other networking issues on their end. But, of course, don't just write it off without investigating as much as you can.

Answer 5

We are experiencing the same "Request timed out" issue with our webservers after switching from IIS6 to IIS7. I believe the issue is IIS7-specific. My guess is that these errors were swallowed or ignored further up the processing chain in IIS6, before the request was handed-off to ASP.Net for processing. I am turning on Failed Request Tracing today to see if I can trap any more information about the issue. So far it seems that your explanation of client-side causes seems the most valid.

Answer 6

I've got the same problem on our production servers that use a small AJAX webservice. After doing packet captures outside our firewall, we found that the POST to the service was coming in two TCP segments and the second segment never got to us. (first packet only contains headers, the second missing packet should be the json body) So basically IIS is just sitting there waiting for the rest of the POST. After the configured timeout, the server sends a RST packet to the client and logs the "Request timed out" error - which is correct behavior.

We are trying to get a client repro without much luck, but in our case this appears to be completely network related (or possibly some "security" software that doesn't like the contents of the post).