How to read the response stream before the Http response completes

Question

When making a request using HttpWebRequest object, I need to call the method GetResponse() to send the request and get the response back.
The problem with this method is that it doesn't return the response object until all data has been received. Say I am downloading a 100 MB file, I won't be able to read it until the response finish and all the 100 MB is downloaded.
What I want is to be able to read the response stream bytes as soon as they arrive, without waiting for the response to complete.
I know I can use the Range Http header, but it won't work on my situation.

Answer 1

If you set the buffer size on your read, you can read in the data in chunks... example...

 // Get the response stream
 using(Stream resStream = response.GetResponseStream())
 {

        string parseString = null;
        int    count      = 0;

        do
        {
            // Read a chunk of data
            count = resStream.Read(buf, 0, buf.Length);

            if (count != 0)
            {
                // Convert to ASCII
                parseString = Encoding.ASCII.GetString(buf, 0, count);

                // Append string to results
                sb.Append(tempString);
            }
        }
        while (count > 0);

}

Answer 2

I think this is very close to what @Zachary suggests. And it (seems to) work(s); actually I think using using as @Zachary does is even "nicer".
My main point being I cannot see the blocking behaviour of GetResponse() you (seem to) describe.

In addition the following code only roughly shows how everything works; it will not read the stream to the end for example (unless by coincidence :)). But it should work if you copy-n-paste it into an empty "Console Application"-project in Visual Studio.

You can try using some "shorter" URL for a test. The example here starts downloading an ISO of the debian distribution (a bit more than 600 MByte). Sorry debian, I did not mean to steal your bandwidth. -> Btw: is there something sensible one can use to test such a scenario?

The Code is strongly inspired by C# - How to read a continuous stream of XML over HTTP.

namespace StreamReadWebRequest
{
    using System;
    using System.Collections.Generic;
    using System.Text;
    using System.Net;
    using System.IO;

    class Program
    {
        static void Main(string[] args)
        {
            HttpWebRequest req;
            HttpWebResponse res = null;

            try
            {
                req = (HttpWebRequest)WebRequest.Create(
                        "http://cdimage.debian.org/debian-cd/5.0.4/i386/iso-cd/debian-504-i386-CD-1.iso");
                res = (HttpWebResponse)req.GetResponse();
                Stream stream = res.GetResponseStream();

                byte[] data = new byte[4096];
                int read;
                while ((read = stream.Read(data, 0, data.Length)) > 0)
                {
                    Process(data, read);
                }
            }
            finally
            {
                if (res != null)
                    res.Close();
            }
            Console.In.Read();
        }

        private static void Process(byte[] data, int read)
        {
            Console.Out.Write(ASCIIEncoding.ASCII.GetString(data));
        }
    }
}

Answer 3

Hello Marwan. I'm not sure what you have on your side, but I know for a fact (and I'm sure many people will agree here) that GetResponse() will NOT download the whole file back. It will send the request, wait for the response, and get the response headers.

After you have the response, you can easily get the response stream with GetResponseStream(), which is the actual data stream that's downloading from the server. And you can easily access the response stream BEFORE the whole file is downloaded. This is 100% true and tested.

If you're not getting the same behaviour (which is really strange, and shouldn't happen) could you add a code example that is not working as I explained above?

Also, do test the example posted by scherand. It just proves once again that it works just fine, without any special hacks.