HttpWebResonse hangs on multiple request

Question

I've an application that create many web request to donwload the news pages of a web site (i've tested for many web sites) after a while I find out that the application slows down in fetching the html source then I found out that HttpWebResonse fails getting the response. I post only the function that do this job.

    public PageFetchResult Fetch()
    {
        PageFetchResult fetchResult = new PageFetchResult();
        try
        {
            HttpWebRequest req = (HttpWebRequest)HttpWebRequest.Create(URLAddress);
            HttpWebResponse resp = (HttpWebResponse)req.GetResponse();
            Uri requestedURI = new Uri(URLAddress);
            Uri responseURI = resp.ResponseUri;
            if (Uri.Equals(requestedURI, responseURI))
            {
                string resultHTML = "";
                byte[] reqHTML = ResponseAsBytes(resp);
                if (!string.IsNullOrEmpty(FetchingEncoding))
                    resultHTML = Encoding.GetEncoding(FetchingEncoding).GetString(reqHTML);
                else if (!string.IsNullOrEmpty(resp.CharacterSet))
                    resultHTML = Encoding.GetEncoding(resp.CharacterSet).GetString(reqHTML);

                resp.Close();
                fetchResult.IsOK = true;
                fetchResult.ResultHTML = resultHTML;
            }
            else
            {
                URLAddress = responseURI.AbsoluteUri;
                relayPageCount++;
                if (relayPageCount > 5)
                {
                    fetchResult.IsOK = false;
                    fetchResult.ErrorMessage = "Maximum page redirection occured.";
                    return fetchResult;
                }
                return Fetch();
            }
        }
        catch (Exception ex)
        {
            fetchResult.IsOK = false;
            fetchResult.ErrorMessage = ex.Message;
        }
        return fetchResult;
    }

any solution would greatly appreciate

Answer 1

Fetch function is called recursively and always creates HttpWebRequest but releasing only when url is matched. You have to close request and response in else statement.

Answer 2

I agree with @volody, Also HttpWebRequest already have property called MaximumAutomaticRedirections, which is set to 50, you can set it to 5 to automatically achieve what you are looking for in this code anyway, it will raise exception and that will be handled by your code.

Just set

request.MaximumAutomaticRedirections  = 5;