Anyone have sample code for doing a "chunked" HTTP streaming download of one web directly to a upload to a separate web server???

Question

Hi,

Background - I'm trying to stream an existing webpage to a separate web application, using HttpWebRequest/HttpWebResponse in C#. One issue I'm striking is that I'm trying to set the file upload request content-length using the file download's content-length, HOWEVER the issue seems to be when the source webpage is on a webserver for which the HttpWebResponse doesn't provide a content length.

HttpWebRequest downloadRequest = WebRequest.Create(new Uri("downloaduri")) as HttpWebRequest;
 using (HttpWebResponse downloadResponse = downloadRequest.GetResponse() as HttpWebResponse)
 {
   var uploadRequest = (HttpWebRequest) WebRequest.Create(new Uri("uripath"));
   uploadRequest.Method = "POST";
   uploadRequest.ContentLength = downloadResponse.ContentLength;  // ####

QUESTION : How could I update this approach to cater for this case (when the download response doesn't have a content-length set). Would it be to somehow use a MemoryStream perhaps? Any sample code would be appreciated. In particular is there a code sample someone would have that shows how to do a "chunked" HTTP download & upload to avoid any issues of the source web server not providing content-length?

Thanks

Answer 1

As I already applied in the Microsoft Forums, there are a couple of options that you have.

However, this is how I would do it with a MemoryStream:

    HttpWebRequest downloadRequest = WebRequest.Create(new Uri("downloaduri")) as HttpWebRequest;

 byte [] buffer = new byte[4096];
 using (MemoryStream ms = new MemoryStream())
 using (HttpWebResponse downloadResponse = downloadRequest.GetResponse() as HttpWebResponse)
 {
   Stream respStream = downloadResponse.GetResponseStream();
   int read = respStream.Read(buffer, 0, buffer.Length);



   while(read > 0)
   {
        ms.Write(buffer, 0, read);
        read = respStream.Read(buffer, 0, buffer.Length);
   }

   // get the data of the stream
   byte [] uploadData = ms.ToArray();

   var uploadRequest = (HttpWebRequest) WebRequest.Create(new Uri("uripath"));
   uploadRequest.Method = "POST";
   uploadRequest.ContentLength = uploadData.Length;


   // you know what to do after this....
}

Also, note that you really dont need to worry about knowing the content-length apriori. As you have guessed, you could have just enabled SendChunked=true on the uploadRequest, and then just copy from the download stream into the upload stream. Or, you can just do the copy without setting "chunked", and HttpWebRequest (AFAIK) will buffer the data internally (make sure AllowWriteStreamBuffering=true on the uploadrequest) and figure out the content length and send the request.