HTTP Headers for Unknown Content-Length

Question

I am currently trying to stream content out to the web after a trans-coding process. This usually works fine by writing binary out to my web stream, but some browsers (specifically IE7, IE8) do not like not having the Content-Length defined in the HTTP header. I believe that "valid" headers are supposed to have this set.

What is the proper way to stream content to the web when you have an unknown Content-Length? The trans-coding process can take awhile, so I want to start streaming it out as it completes.

Answer 1

Try sending them in chunks along with Transfer-Encoding: chunked. More details in wikipedia.

Update as per the comments, here's an example how a "ChunkedOutputStream" in Java may look like:

package com.stackoverflow.q2395192;

import java.io.IOException;
import java.io.OutputStream;

public class ChunkedOutputStream extends OutputStream {

    private static final byte[] CRLF = "\r\n".getBytes();
    private OutputStream output = null;

    public ChunkedOutputStream(OutputStream output) {
        this.output = output;
    }

    @Override
    public void write(int i) throws IOException {
        write(new byte[] { (byte) i }, 0, 1);
    }

    @Override
    public void write(byte[] b, int offset, int length) throws IOException {
        writeHeader(length);
        output.write(CRLF, 0, CRLF.length);
        output.write(b, offset, length);
        output.write(CRLF, 0, CRLF.length);
    }

    @Override
    public void flush() throws IOException {
        output.flush();
    }

    @Override
    public void close() throws IOException {
        writeHeader(0);
        output.write(CRLF, 0, CRLF.length);
        output.write(CRLF, 0, CRLF.length);
        output.close();
    }

    private void writeHeader(int length) throws IOException {
        byte[] header = Integer.toHexString(length).getBytes();
        output.write(header, 0, header.length);
    }

}

...which can basically be used as:

OutputStream output = new ChunkedOutputStream(response.getOutputStream());
output.write(....);

You see in the source, every chunk of data exist of a header which represents the length of data in hex, a CRLF, the actual data and a CRLF. The end of the stream is represented by a header denoting a 0 length and two CRLFs.

Note: despite the example, you actually do not need it in a JSP/Servlet based webapplication. Whenever the content length is not set on a response, the webcontainer will automatically transfer them in chunks.

Answer 2

Just as a follow up to BalusC's excellent post, here is the code I am using in C#. I am chunking data manually directly to an HTTP output stream, after receiving data from the STDOUT on a process.

int buffSize = 16384;
byte[] buffer = new byte[buffSize];
byte[] hexBuff;
byte[] CRLF = Encoding.UTF8.GetBytes("\r\n");

br = new BinaryReader(transcoder.StandardOutput.BaseStream);

//Begin chunking...
int ret = 0;
while (!transcoder.HasExited && (ret = br.Read(buffer, 0, buffSize)) > 0)
{
    //Write hex length...
    hexBuff = Encoding.UTF8.GetBytes(ret.ToString("X"));
    e.Context.Stream.Write(hexBuff, 0, hexBuff.Length);

    //Write CRLF...
    e.Context.Stream.Write(CRLF, 0, CRLF.Length);

    //Write byte content...
    e.Context.Stream.Write(buffer, 0, ret);

    //Write CRLF...
    e.Context.Stream.Write(CRLF, 0, CRLF.Length);
}
//End chunking...
//Write hex length...
hexBuff = Encoding.UTF8.GetBytes(0.ToString("X"));
e.Context.Stream.Write(hexBuff, 0, hexBuff.Length);