Chunked Encoding Implementation in .NET (or at least pseudo code)

Question

I wrote a raw TCP client for HTTP/HTTPS requests, however I'm having problems with chunked encoding responses. HTTP/1.1 is requirement therefore I should support it.

Raw TCP is a business requirement that I need to keep, therefore I can't switch to .NET HTTPWebRequest/HTTPWebResponse However if there is way to convert a RAW HTTP Request/Response into HTTPWebRequest/HTTPWebResponse that'd work.

Answer 1

The best place to start is the http 1.1 specification, which lays out how chunking works. Specifically section 3.6.1.

3.6.1 Chunked Transfer Coding

The chunked encoding modifies the body of a message in order to
transfer it as a series of chunks, each with its own size indicator,
followed by an OPTIONAL trailer containing entity-header fields. This allows dynamically produced content to be transferred along with the
information necessary for the recipient to verify that it has
received the full message.

   Chunked-Body   = *chunk
                    last-chunk
                    trailer
                    CRLF

   chunk          = chunk-size [ chunk-extension ] CRLF
                    chunk-data CRLF
   chunk-size     = 1*HEX
   last-chunk     = 1*("0") [ chunk-extension ] CRLF

   chunk-extension= *( ";" chunk-ext-name [ "=" chunk-ext-val ] )
   chunk-ext-name = token
   chunk-ext-val  = token | quoted-string
   chunk-data     = chunk-size(OCTET)
   trailer        = *(entity-header CRLF)

The chunk-size field is a string of hex digits indicating the size of
the chunk. The chunked encoding is ended by any chunk whose size is
zero, followed by the trailer, which is terminated by an empty line.

The trailer allows the sender to include additional HTTP header
fields at the end of the message. The Trailer header field can be used to indicate which header fields are included in a trailer (see section 14.40).

Assuming that you have already read the header from the response and are pointing to the next byte in the stream your pseudo code would look something like this:

done = false;
uint8 bytes[];
while (!done)
{
  chunksizeString = readuntilCRLF(); // read in the chunksize as a string
  chunksizeString.strip(); // strip off the CRLF
  chunksize = chunksizeString.convertHexString2Int(); // convert the hex string to an integer.
  bytes.append(readXBytes(chunksize)); // read in the x bytes and append them to your buffer.
  readCRLF(); // read the trailing CRLF and throw it away.
  if (chunksize == 0)
     done = true; //

}
// now read the trailer if any
// trailer is optional, so it may be just the empty string
trailer = readuntilCRLF()
trailer = trailer.strip()
if (trailer != "")
   readCRLF(); // read out the last CRLF and we are done.

This is ignoring the chunk-extension portion, but since it is delimited with a ";" it should be easy to split it out. This should be enough to get you started. Keep in mind that the chunksize string does not have a leading "0x".

Answer 2

For the future reference also I found this :

 length := 0
   read chunk-size, chunk-extension (if any) and CRLF
   while (chunk-size > 0) {
      read chunk-data and CRLF
      append chunk-data to entity-body
      length := length + chunk-size
      read chunk-size and CRLF
   }
   read entity-header
   while (entity-header not empty) {
      append entity-header to existing header fields
      read entity-header
   }
   Content-Length := length
   Remove "chunked" from Transfer-Encoding