Internet Explorer 8 + Deflate

Question

I have a very weird problem.. I really do hope someone has an answer because I wouldn't know where else to ask.

I am writing a cgi application in C++ which is executed by Apache and outputs HTML code. I am compressing the HTML output myself - from within my C++ application - since my web host doesn't support mod_deflate for some reason.

I tested this with Firefox 2, Firefox 3, Opera 9, Opera 10, Google Chrome, Safari, IE6, IE7, IE8, even wget.. It works with ANYTHING except IE8.

IE8 just says "Internet Explorer cannot display the webpage", with no information whatsoever. I know it's because of the compression only because it works if I disable it.

Do you know what I'm doing wrong?

I use zlib to compress it, and the exact code is:

    /* Compress it */
int compressed_output_size = content.length() + (content.length() * 0.2) + 16;
char *compressed_output = (char *)Alloc(compressed_output_size);
int compressed_output_length;
Compress(compressed_output, compressed_output_size, (void *)content.c_str(), content.length(), &compressed_output_length);

/* Send the compressed header */
cout << "Content-Encoding: deflate\r\n";
cout << boost::format("Content-Length: %d\r\n") % compressed_output_length;
cgiHeaderContentType("text/html");
cout.write(compressed_output, compressed_output_length);


static void Compress(void *to, size_t to_size, void *from, size_t from_size, int *final_size)
{
int ret;
z_stream stream;

stream.zalloc = Z_NULL;
stream.zfree = Z_NULL;
stream.opaque = Z_NULL;

if ((ret = deflateInit(&stream, CompressionSpeed)) != Z_OK)
 COMPRESSION_ERROR("deflateInit() failed: %d", ret);

stream.next_out = (Bytef *)to;
stream.avail_out = (uInt)to_size;
stream.next_in = (Bytef *)from;
stream.avail_in = (uInt)from_size;

if ((ret = deflate(&stream, Z_NO_FLUSH)) != Z_OK)
 COMPRESSION_ERROR("deflate() failed: %d", ret);

if (stream.avail_in != 0)
 COMPRESSION_ERROR("stream.avail_in is not 0 (it's %d)", stream.avail_in);

if ((ret = deflate(&stream, Z_FINISH)) != Z_STREAM_END)
 COMPRESSION_ERROR("deflate() failed: %d", ret);

if ((ret = deflateEnd(&stream)) != Z_OK)
 COMPRESSION_ERROR("deflateEnd() failed: %d", ret);

if (final_size)
 *final_size = stream.total_out;
return;
}

Answer 1

The gzip and deflate methods aren't the same... they are very close, but there are some subtle differences with the header, so, if you change your content-encoding, you should also change your parameters to the encoding method (specifically, the window size)!

See: http://apcmag.com/improve_your_site_with_http_compression.htm

Probably the other browsers are ignoring your content-encoding specification and doing some automatic recognition, but IE8 is not...

See: http://www.zlib.net/manual.html#deflateInit2

Try to use:

method=Z_DEFLATED
windowBits=-15  (negative so that the header is suppressed)

And use "gzip" as the content-encoding

Answer 2

I wanted to clarify what I've discovered on this, as I've written my own deflate algorithm, my own HTTP server, and to my dismay IE8 also failed to recognize my deflated content:

HTTP RFC is http://www.faqs.org/ftp/rfc/rfc2616.pdf. Page 17 states both RFC 1950 and RFC 1951 is used when performing a deflate in the HTTP headers. RFC 1950 is simply defining the header and trailer bytes; the deflate algorithm is defined in RFC 1951. When I programmed this to spec, IE8 failed.

When I ignored RFC 1950 and only did RFC 1951, it passed.

I would assume, then, that IE8 isn't following RFC 2616 page 17 correctly, and all of the other browsers are nice enough to accept either format.