calculating filesize of a response?

Question

hello, im tryingto code a script to do dynamic transcoding of video and audio (likely just audio for now) and streaming to mobile devices, currently im using the script posted elsewhere on here (http://stackoverflow.com/questions/1608571/how-do-you-convert-audio-files-on-the-fly-to-the-browser) but this seemsto have problems as i do not know how to calculate the filesize of the outputted data, can anyone suggest how i might calculate this dynamicly and pass the correct header?

Answer 1

(Updated with final solution to specific problem at bottom of answer.)

You can't

Audio and video encoding is, in this case (and most others), not predictable when it comes to the exact resulting file size. Remember the fact that your source material's characteristic and impact on the final file size cannot be predicted while encoding either. If you are transcoding 'on the fly', you are doing two things 1) compressing, 2) losing data. It just cannot be done with an mp3 output format.

You might...if: the encoding was lossless, and your algorithm was tuned to very specific source material characteristics (sample rate, sample size, etc.) and both the source and target formats where uncompressed. But that is not your situation.

As for the header: don't sent it! The Content-Length header is not an HTTP 1.1 requirement. There are disadvantages to this (progress bars cannot ever know what 100% done until end-of-file; no 'time remaining' is possible either) but my guess is you can live without it.

---

Final discussion based on comments:

With a browser, I get the behavior you are describing. And with with this curl command (helpful for debugging low level pain like this), it also does not work:

curl --trace-ascii trace0.txt "http://dmpwap.net/playmp3.php?b=128&file=Red_Hot_Chili_Peppers_-_15_-_Fortune_Faded.mp3" > test0.mp3

I get 0 bytes, and see this in my trace:

manoa:~ stu$ cat trace0.txt == Info: About to connect() to dmpwap.net port 80 (#0)
== Info:   Trying 64.191.50.69... == Info: connected
== Info: Connected to dmpwap.net (64.191.50.69) port 80 (#0)
=> Send header, 213 bytes (0xd5)
0000: GET /playmp3.php?b=128&file=Red_Hot_Chili_Peppers_-_15_-_Fortune
0040: _Faded.mp2 HTTP/1.1
0055: User-Agent: curl/7.19.4 (universal-apple-darwin10.0) libcurl/7.1
0095: 9.4 OpenSSL/0.9.8k zlib/1.2.3
00b4: Host: dmpwap.net
00c6: Accept: */*
00d3: 
<= Recv header, 17 bytes (0x11)
0000: HTTP/1.1 200 OK
<= Recv header, 19 bytes (0x13)
0000: Connection: close
<= Recv header, 37 bytes (0x25)
0000: Date: Fri, 11 Dec 2009 14:04:58 GMT
<= Recv header, 27 bytes (0x1b)
0000: Server: Microsoft-IIS/6.0
<= Recv header, 27 bytes (0x1b)
0000: X-Powered-By: PHP/5.2.9-2
<= Recv header, 35 bytes (0x23)
0000: Content-Transfer-Encoding: binary
<= Recv header, 26 bytes (0x1a)
0000: Content-Type: audio/mpeg
<= Recv header, 2 bytes (0x2)
0000: 
<= Recv data, 0 bytes (0x0)
== Info: Closing connection #0

But....

If I add a --tcp-nodelay option, it works just fine! E.g.:

curl --tcp-nodelay --trace-ascii trace1.txt "http://dmpwap.net/playmp3.php?b=128&file=Red_Hot_Chili_Peppers_-_15_-_Fortune_Faded.mp3" > test1.mp3

It returned 3219104 bytes. The trace.txt looks like this:

== Info: About to connect() to dmpwap.net port 80 (#0)
== Info:   Trying 64.191.50.69... == Info: TCP_NODELAY set
== Info: connected
== Info: Connected to dmpwap.net (64.191.50.69) port 80 (#0)
=> Send header, 213 bytes (0xd5)
0000: GET /playmp3.php?b=128&file=Red_Hot_Chili_Peppers_-_15_-_Fortune
0040: _Faded.mp3 HTTP/1.1
0055: User-Agent: curl/7.19.4 (universal-apple-darwin10.0) libcurl/7.1
0095: 9.4 OpenSSL/0.9.8k zlib/1.2.3
00b4: Host: dmpwap.net
00c6: Accept: */*
00d3: 
<= Recv header, 17 bytes (0x11)
0000: HTTP/1.1 200 OK
<= Recv header, 19 bytes (0x13)
0000: Connection: close
<= Recv header, 37 bytes (0x25)
0000: Date: Fri, 11 Dec 2009 13:56:47 GMT
<= Recv header, 27 bytes (0x1b)
0000: Server: Microsoft-IIS/6.0
<= Recv header, 27 bytes (0x1b)
0000: X-Powered-By: PHP/5.2.9-2
<= Recv header, 35 bytes (0x23)
0000: Content-Transfer-Encoding: binary
<= Recv header, 26 bytes (0x1a)
0000: Content-Type: audio/mpeg
<= Recv header, 2 bytes (0x2)
0000: 
<= Recv data, 1258 bytes (0x4ea)
0000: ID3.......TENC.......Lavf52.23.1...d.... ..=....w......oq......0
   ...  {many lines}
0180: UUUUUU
== Info: Closing connection #0

I can listen to the song (3m21s, stereo, mpga, 48kHz, 128kbps) with no problems.

So, my theory is, that because there are consecutive 0x00 bytes in the stream, the clients are thinking "OK, I got 0x00, 0x00...and nothing else has been sent, the connection must be over." But with the --tcp-nodelay option set on the curl client, it does not happen.

My candidate solution: Disable Nagel's algorithm (set TCP no delay on socket options) server side, at least for these transcoding request connections. This will prevent buffering, which I suspect is leading to the connections being dropped.