There's a binary file on a remote server, that I wish to stream to my client. I issue a GET request and the response is an HTTP header + body that has the binary file, in chunks. The problem is, each chunk contains as well as the binary data, some metadata that I need.
How can I read just ONE chunk at a time from the HTTP stream with C#?
Ani's solution isn't going to cut it for you, because your problem isn't about streaming it in, but rather getting access to the original, protocol-level chunks. Unless the metadata is added to the returned headers, which I suspect is not the case, your only choice might be to drop down to the socket level and implement your own HTTP client code. Depending on how mich you can constrain the functionality and your own comfort with protocols, this might not be so hard.
edit
If you opened up a socket, sent a well-formed HTTP 1.1 request and then read everything back, you would find the initial response headers followed by some number of chunks. Each chunk has its mini-header, followed by the associated data. If you want information in those mini-headers, it's there for you, but you'd have to parse it all out for yourself, just as you'd have to parse the response header. These details are all handled for you when you use a high-level protocol class, but then again, the details are also hidden from you, which is a problem for your special needs.
Now, if you were to read up to and including the first chunk header, it would contain the length of that chunk, so you'd know exactly how many bytes you can read until the end of the chunk. I believe, but I'm not certain, that if you tried to read past the end of that chunk and the next chunk wasn't sent, Socket.Receive would read as much as it can and return the actual byte count. In any case, if you were careful, you would be able to start processing the first chunk while the second was still being sent.
Does that help?
As others have pointed out, it is not possible to issue a Read() and expect to get back one chunk of data. HTTP uses TCP which is a streaming protocol. THis means that if the sender wrote 1024 bytes, then the reader counld get the daata in 1024 reads of 1 byte, or 1 read of 1024 bytes or anything in between.
Also due to this reason, even going down to Sockets wont help, since the underlying protocol is still TCP.
SO, you have to do it the hard way. You will have to write a state machine that first does enough Read() to get the required amount of bytes that contain the metadata. Parse this byte array and get the metadata. Then, buffer the remaining (if any) and continue reading to get the Data part. Lather, rinse, repeat...
EDIT:
In order to describe what we mean by "parsing it out yourself" have a look at this article that describes how to read and parse an employee record from the wire data. You can look at the .NET Read(TcpClient) method. This will give you a good idea on how to implement your routine.