Showing a long running shell process with Apache

Question

I have a CGI script which takes about 1 minute to run. Right now Apache only returns results to the browser once the process has finished.

How can I make it show the output like it was run on a terminal?

Here is a example which demonstrates the problem.

I want to see the numbers 1 to 5 appear as they are printed.

Answer 1

There are several factors at play here. To eliminate a few issues, Apache and bash are not buffering any of the output. You can verify with this script:

#!/bin/sh

cat <<END
Content-Type: text/plain

END

for i in {1..10}
do
    echo $i
    sleep 1
done

Stick this somewhere that Apache is configured to execute CGI scripts, and test with netcat:

$ nc localhost 80
GET /cgi-bin/chunkit.cgi HTTP/1.1
Host: localhost

HTTP/1.1 200 OK
Date: Tue, 24 Aug 2010 23:26:24 GMT
Server: Apache/2.2.14 (Unix) mod_ssl/2.2.14 OpenSSL/0.9.7l DAV/2
Transfer-Encoding: chunked
Content-Type: text/plain

2
1

2
2

2
3

2
4

2
5

2
6

2
7

2
8

2
9

3
10

0

When I do this, I see in netcat each number appearing once per second, as intended.

Note that my version of Apache, at least, applies the chunked transfer encoding automatically, presumably because I didn't include a Content-Length; if you return the Transfer-Encoding: chunked header yourself, then you need to encode the output of your script in the chunked transfer encoding. That's pretty easy, even in a shell script:

chunk () {
    printf '%x\r\n' "${#1}"  # Length of the chunk in hex, CRLF
    printf '%s\r\n' "$1"     # Chunk itself, CRLF
}

chunk $'1\n' # This is a Bash-ism, since it's pretty hard to get a newline
chunk $'2\n' # character portably.

However, serve this to a browser, and you'll get varying results depending on the browser. On my system, Mac OS X 10.5.8, I see different behaviors between my browsers. In Safari, Chrome, and Firefox 4 beta, I don't start seeing output until I've sent somewhere around 1000 characters (I would guess 1024 including the headers, or something like that, but I haven't narrowed it down to the exact behavior). In Firefox 3.6, it starts displaying immediately.

I would guess that this delay is due to content type sniffing, or character encoding sniffing, which are in the process of being standardized. I have tried to see if I could get around the delay by specifying proper content types and character encodings, but without luck. You may have to send some padding data (which would be pretty easy to do invisibly if you use HTML instead of plain text), to get beyond that initial buffer.

Once you start streaming HTML instead of plain text, the structure of your HTML matters too. Some content can be displayed progressively, while some cannot. For instance, streaming down <div>s into the body, with no styling, works fine, and can display progressively as it arrives. If you try to open a <pre> tag, and just stream content into that, Webkit based browsers will wait until they see the close tag to try to lay that out, while Firefox is happy to display it progressively. I don't know all of the corner cases; you'll have to experiment to see what works for you.

Anyhow, I hope this helps you get started. Let me know if you have any more questions!