Is there a way that I can force mod_perl to re-use buffer memory?

Question

Hi,

I have a Perl script running in mod_perl that needs to write a large amount of data to the client, possibly over a long period. The behavior that I observe is that once I print and flush something, the buffer memory is not reclaimed even though I rflush (I know this can't be reclaimed back by the OS).

Is that how mod_perl operates and is there a way that I can force it to periodically free the buffer memory, so that I can use that for new buffers instead of taking more from the OS?

Just to clarify, I'm not using any buffers myself and there are no leaks in my code. Consider the following simple example:

  sub handler { 
     my $request = shift; 
     my $boundary = time; 
     $request->content_type("multipart/x-mixed-replace;boundary=\"$boundary\";"); 
     for (;;) { 
        $request->print("--$boundary\n"); 
        $request->print("Content-type: text/html; charset=utf-8;\n\n"); 
        $request->print("$data\n\n"); 
        $request->rflush;
     } 
     return Apache2::Const::OK; 
  } 

This leaks heavily, and my request is being kept alive, so it may be active for days.

Answer 1

Release any references you have to the buffer. For example, if you were using a string as a buffer as in

$buf = "really long string " . "and other methods that make it huger";
print SOMEWHERE $buf;
$buf = ""; # or undef $buf according to taste

Should return the storage formerly pointed to by $buf to the free pool.

Answer 2

Not returning memory to the OS is the standard behavior of the perl interpreter itself, not specific to mod_perl per se. Other than using shared memory (which, IIRC, you handle the allocation/de-allocation for manually) or terminating the process, I'm not aware of any way of getting perl to release memory back to the host OS.

Letting variables pass out of scope will allow perl to re-use that memory for other variables, but will not return it to the OS.

Edit: I just re-read the question and realized that you're just looking for a way to let perl re-use the memory, not attempting to release it to the OS. In that case, using lexical (my) variables and confining them to the smallest possible scope instead of defining a global buffer early and keeping it around forever should do the trick.

Answer 3

Your for(;;) loop can never end as-written, which will lead to worse problems than leaking memory. The print method must allocate some memory, probably as part of the request record, which is normally released when the request is cleaned up. This is happening in the C code, either within mod_perl2 or Apache2.

You'll have to redesign your approach to get around this. Instead of sending the long-running response from within the mod_perl handler, redirect the user through a ProxyPass setting to a program that will print the response to STDOUT. (Essentially a CGI script.) The script can be pure-perl where the techniques other posters have mentioned about restricting scope of variables will work. The response will still pass through Apache, but when operating as a reverse proxy Apache has a fixed set of buffers that it copies the data through in a bucket-brigade; I've never seen my reverse proxy processes consume much memory despite passing through huge amounts of data.