Perl map/grep memory leak

Question

I have been working on a perl project at work, and came across a strange memory leak. I have boiled down the source of my problem into a contrived example:

#!/usr/bin/perl
use strict;
use warnings;

# takes: an array reference
# returns: 1
sub g {
    my ($a) = @_;
    return 1; 
}

# takes: nothing
# returns: the result of applying g on an array reference
sub f {
    my @a = ('a') x 131072; # allocate roughly a megabyte 
    return g(\@a); 
}

# causes a leak:
#map { f($_) } (1..100000); 

# loop equivalent to map, no leak:
#my @b;
#for my $i (1..100000) {
#    push @b, f($_);
#}

# causes a leak:
#grep { f($_) } (1..100000);

# loop equivalent to grep, no leak:
#my @b;
#for my $i (1..100000) {
#    push @b, $i if f($_);
#}

Uncomment 1 of the 4 blocks of code (beneath the subroutines) at a time and run the script while monitoring its memory usage. On my machine, the code that uses grep or map appear to cause memory leaks, whereas the "loop equivalent"s do not. My perl version is v5.10.1, and I am running Ubuntu.

I believe this could be a bug in perl, but I don't want to jump to a drastic conclusion without another opinion on what could be the cause. Can anyone explain if this behaviour is correct?

Thanks

Answer 1

I don't know if it is a memory leak as such. If I lower the top value of your loop (say, from 100000 to 100), I can use the map/grep expressions repeatedly without losing memory.

Rather, it seems more likely that map and grep are atomic operations when it comes to memory management, that perl is not performing its garbage collection in the middle of those operations.

Perl 5.12.0 (and 5.8.9) do seem a bit more robust on these kind of expressions (but they also seem to be slower).

Answer 2

It really is. But to prove so, you have to put while(1){} around suspicious expression - in perl, memory that is once acquired never returns to OS (but can be reused by perl itself). I've run code with

while(1){ grep { f($_) } (1..100000) }

under 5.8.8 and it constantly grows in size - so, it's a leak.