System.Net.WebException: The underlying connection was closed: An unexpected error occurred on a receive.

Question

I'm trying to create a method in C# to return a string of a web pages html content from the url. I have tried several different ways, but I am getting the error System.Net.WebException: The underlying connection was closed: An unexpected error occurred on a receive.

The following works fine locally, but gets the above error when running on a remote server:

  public static string WebPageRead(string url)
    {
        string result = String.Empty;

        WebResponse response = null;
        StreamReader reader = null;

        try
        {
            if (!String.IsNullOrEmpty(url))
            {
                HttpWebRequest request = HttpWebRequest.Create(url) as HttpWebRequest;
                request.Method = "GET";
                request.KeepAlive = false;
                request.ProtocolVersion = HttpVersion.Version10;

                response = request.GetResponse();
                reader = new StreamReader(response.GetResponseStream(), Encoding.UTF8);
                result = reader.ReadToEnd();
            }
        }
        catch (Exception exc)
        {
            throw exc;
        }
        finally
        {
            if (reader != null)
            {
                reader.Close();
            }

            if (response != null)
            {
                response.Close();
            }
        }

        return result;
    }

Answer 1

Had a problem like this before that was solved by opening the url in IE on the machine with the problem. IE then asks you whether you want to add the url to the list of secure sites. Add it and it works for that url.

This is just one of the possible causes. Seriously a lot of other problems could cause this. Besides the problem described above, the best way I've found to solve this is the just catch the exception and retry the request.

Answer 2

This is probably not the problem, but try the following:

public static string WebPageRead(string url)
{
    if (String.IsNullOrEmpty(url))
    {
        return null;
    }

    HttpWebRequest request = HttpWebRequest.Create(url) as HttpWebRequest;
    if (request == null)
    {
        return null;
    }

    request.Method = "GET";
    request.KeepAlive = false;
    request.ProtocolVersion = HttpVersion.Version10;

    using (WebResponse response = request.GetResponse())
    {
        using (Stream stream = response.GetResponseStream())
        {
            using (StreamReader reader = 
                       new StreamReader(stream, Encoding.UTF8))
            {
                return reader.ReadToEnd();
            }
        }
    }
}

I echo the earlier answer that suggests you try this with a known good URL. I'll add that you should try this with a known good HTTP 1.1 URL, commenting out the line that sets the version to 1.0. If that works, then it narrows things down considerably.

Answer 3

Thanks for the responses, the problem was due to a DNS issue on the remote server! Just to confirm, I went with the following code in the end:

    public static string WebPageRead(string url)
    {
        string content = String.Empty;

        if (!String.IsNullOrEmpty(url))
        {
            HttpWebRequest request = HttpWebRequest.Create(url) as HttpWebRequest;

            if (request != null)
            {
                request.Method = "GET";
                request.KeepAlive = false;
                request.ProtocolVersion = HttpVersion.Version10;

                try
                {
                    using (WebResponse response = request.GetResponse())
                    {
                        using (Stream stream = response.GetResponseStream())
                        {
                            using (StreamReader reader = new StreamReader(stream, Encoding.UTF8))
                            {
                                content = reader.ReadToEnd();
                            }
                        }
                    }
                }
                catch (Exception exc)
                {
                    throw exc;
                }
            }
        }                    

        return content;
    }