ASP.NET HttpWebRequest stop sending requests, high CPU usage

Question

I have an old ASP.NET application. We send out httpWebRequest to a remote REST server and retrieve XML back, most of the time the application works fine. Recently, we got some high CPU usage issue several times a day.

During the high CPU usage time, we monitored those httpWebRequest connections (by checking netstat for the w3wp process). At the very beginning, the connections change to "CLOSE_WAIT" status from "ESTABLISHED", then after those connections timeout, those connections disappeared one by one, and then there is no connection any more.

After resetting the IIS, when the w3wp.exe process start again, we still could not find any connections to httpWebRequest target server. So the CPU usage keep staying at high level. Even after several round of reset, it won't solve the issue, until we saw some connections start to connect to httpWebRequest target server, the CPU usage went down.

I actually thought it could be the issue of my code not handling the httpWebRequest properly, I posted another question here: How to close underlying connections after catch httpwebrequest timeout?.

As mentioned in that question, I also found lots of timeout exceptions for System.Net.HttpWebRequest.GetResponse(). We found 3500 of the same exception within 5 minutes when CPU usage is really high.

What could cuase this type of issue and what could be the medicine? Why won't the application send out request any more (since there is no connection in netstat)?

Here is the source code just in case:

System.Net.HttpWebResponse httpWebResponse = null;
System.IO.Stream stream  = null;
XmlTextReader xmlTextReader  = null;
try
{
    System.Net.HttpWebRequest httpWebRequest = (System.Net.HttpWebRequest)System.Net.WebRequest.Create(request);
    httpWebRequest.ReadWriteTimeout = 10000;
    httpWebRequest.Timeout = 10000;
    httpWebRequest.KeepAlive = false;
    httpWebRequest.Method = "GET";
    httpWebResponse = (System.Net.HttpWebResponse)httpWebRequest.GetResponse();
    stream = httpWebResponse.GetResponseStream();
    xmlTextReader = new  XmlTextReader(stream);
    xmlTextReader.Read();
    xmlDocument.Load(xmlTextReader);
    //Document processing code.
    //...
}
catch
{
    //Catch blcok with error handle
}
finally
{
    if (xmlTextReader != null)
        xmlTextReader.Close();
    if (httpWebResponse != null)
        httpWebResponse.Close();
    if (stream != null)
        stream.Close();
}

Answer 1

I would suggest you to try sending requests asynchronously to avoid blocking the main thread:

using (var client = new WebClient())
{
    client.OpenReadCompleted += (sender, e) =>
    {
        using (var reader = XmlReader.Create(e.Result))
        {
            // Process the XML document here
        }
    };
    client.OpenReadAsync(new Uri("http://www.example.com"));
}

Answer 2

From your description, it is not clear to me that your high CPU utilization is related to your outgoing HTTP requests. High CPU utilization could be due to a bug in your code, a bug in CLR, IIS, or something else. Without knowing which component is consuming the CPU, you wont be able to do anything further.

If I were you, I would first try to attach a sampling profiler to the W3WP process, and see which component is consuming the CPU. That should point you to the next steps in resolving this issue.