Compressing HTTP request with LWP, Apache, and mod_deflate

Question

I have a client/server system that performs communication using XML transferred using HTTP requests and responses with the client using Perl's LWP and the server running Perl's CGI.pm through Apache. In addition the stream is encrypted using SSL with certificates for both the server and all clients.

This system works well, except that periodically the client needs to send really large amounts of data. An obvious solution would be to compress the data on the client side, send it over, and decompress it on the server. Rather than implement this myself, I was hoping to use Apache's mod_deflate's "Input Decompression" as described here.

The description warns:

If you evaluate the request body yourself, don't trust the Content-Length header! The Content-Length header reflects the length of the incoming data from the client and not the byte count of the decompressed data stream.

So if I provide a Content-Length value which matches the compressed data size, the data is truncated. This is because mod_deflate decompresses the stream, but CGI.pm only reads to the Content-Length limit.

Alternatively, if I try to outsmart it and override the Content-Length header with the decompressed data size, LWP complains and resets the value to the compressed length, leaving me with the same problem.

Finally, I attempted to hack the part of LWP which does the correction. The original code is:

    # Set (or override) Content-Length header
    my $clen = $request_headers->header('Content-Length');
    if (defined($$content_ref) && length($$content_ref)) {
        $has_content = length($$content_ref);
        if (!defined($clen) || $clen ne $has_content) {
            if (defined $clen) {
                warn "Content-Length header value was wrong, fixed";
                hlist_remove(\@h, 'Content-Length');
            }
            push(@h, 'Content-Length' => $has_content);
        }
    }
    elsif ($clen) {
        warn "Content-Length set when there is no content, fixed";
        hlist_remove(\@h, 'Content-Length');
    }

And I changed the push line to:

  push(@h, 'Content-Length' => $clen);

Unfortunately this causes some problem where content (truncated or not) doesn't even get to my CGI script.

Has anyone made this work? I found this which does compression on a file before uploading, but not compressing a generic request.

Answer 1

I am not sure if I am following you with what you want, but I have a custom get/post module, that I use to do some non standard stuff. The below code will read in anything sent via post, or STDIN.

read(STDIN, $query_string, $ENV{'CONTENT_LENGTH'});

Instead of using using $ENV's value use your's. I hope this helps, and sorry if it doesn't.

Answer 2

I don't think you can change the Content-Length like that. It would confuse Apache, because mod_deflate wouldn't know how much compressed data to read. What about having the client add an X-Uncompressed-Length header, and then use a modified version of CGI.pm that uses X-Uncompressed-Length (if present) instead of Content-Length? (Actually, you probably don't need to modify CGI.pm. Just set $ENV{'CONTENT_LENGTH'} to the appropriate value before initializing the CGI object or calling any CGI functions.)

Or, use a lower-level module that uses the bucket brigade to tell how much data to read.

Answer 3

Although you said you didn't want to do the compression yourself, there are lots of perl modules which will do both sides for you, Compress::Zlib for example.

I have a cheat (with a .net part of the company) where I get passed XML as a separate parameter posted in, then can handle it as if it was a string rather than faffing about with SOAP like stuff.