C# - WebRequest Doesn't Return Different Pages

Question

Here's the purpose of my console program: Make a web request > Save results from web request > Use QueryString to get next page from web request > Save those results > Use QueryString to get next page from web request, etc.

So here's some pseudocode for how I set the code up.

 for (int i = 0; i < 3; i++)
        {
            strPageNo = Convert.ToString(i);  

            //creates the url I want, with incrementing pages
            strURL = "http://www.website.com/results.aspx?page=" + strPageNo;   

            //makes the web request
            wrGETURL = WebRequest.Create(strURL);

            //gets the web page for me
            objStream = wrGETURL.GetResponse().GetResponseStream();

            //for reading web page
            objReader = new StreamReader(objStream);

            //--------
            // -snip- code that saves it to file, etc.
            //--------

            objStream.Close();
            objReader.Close();

            //so the server doesn't get hammered
            System.Threading.Thread.Sleep(1000); 
         }

Pretty simple, right? The problem is, even though it increments the page number to get a different web page, I'm getting the exact same results page each time the loop runs.

i IS incrementing correctly, and I can cut/paste the url strURL creates into a web browser and it works just fine.

I can manually type in &page=1, &page=2, &page=3, and it'll return the correct pages. Somehow putting the increment in there screws it up.

Does it have anything to do with sessions, or what? I make sure I close both the stream and the reader before it loops again...

Answer 1

That URL doesn't quite make sense to me unless you are using MVC or something that can interpret the querystring correctly.

http://www.website.com/results.aspx&page=

should be:

http://www.website.com/results.aspx?page=

Some browsers will accept poorly formed URLs and render them fine. Others may not which may be the problem with your console app.

Answer 2

Have you tried creating a new WebRequest object for each time during the loop, it could be the Create() method isn't adequately flushing out all of its old data.

Another thing to check is that the ResponseStream is adequately flushed out before the next loop iteration.

Answer 3

Just a suggestion, try disposing the Stream, and the Reader. I've seen some weird cases where not disposing objects like these and using them in loops can yield some wacky results....

Answer 4

This code works fine for me:

var urls = new [] { "http://www.google.com", "http://www.yahoo.com", "http://www.live.com" };

foreach (var url in urls)
{
    WebRequest request = WebRequest.Create(url);
    using (Stream responseStream = request.GetResponse().GetResponseStream())
    using (Stream outputStream = new FileStream("file" + DateTime.Now.Ticks.ToString(), FileMode.Create, FileAccess.Write, FileShare.None))
    {
        const int chunkSize = 1024;
        byte[] buffer = new byte[chunkSize];
        int bytesRead;
        while ((bytesRead = responseStream.Read(buffer, 0, buffer.Length)) > 0)
        {
            byte[] actual = new byte[bytesRead];
            Buffer.BlockCopy(buffer, 0, actual, 0, bytesRead);
            outputStream.Write(actual, 0, actual.Length);
        }
    }
    Thread.Sleep(1000);
}

Answer 5

Here's my terrible, hack-ish, workaround solution:

Make another console app that calls THIS one, in which the first console app passes an argument at the end of strURL. It works, but I feel so dirty.