In Perl, how can I release memory to the operating system?

Question

I am having some problems with memory in Perl. When I fill up a big hash, I can not get the memory to be released back to the OS. When I do the same with a scalar and use undef, it will give the memory back to the OS.

Here is a test program I wrote.

#!/usr/bin/perl
###### Memory test
######

## Use Commands
use Number::Bytes::Human qw(format_bytes);
use Data::Dumper;
use Devel::Size qw(size total_size);

## Create Varable
my $share_var;
my %share_hash;
my $type_hash = 1;
my $type_scalar = 1;

## Start Main Loop
while (true) {
    &Memory_Check();
    print "Hit Enter (add to memory): "; <>;
    &Up_Mem(100_000);
    &Memory_Check();

    print "Hit Enter (Set Varable to nothing): "; <>;
    $share_var = "";
    $share_hash = ();
    &Memory_Check();

    print "Hit Enter (clean data): "; <>;
    &Clean_Data();
    &Memory_Check();

    print "Hit Enter (start over): "; <>;
}

exit;


#### Up Memory
sub Up_Mem {
    my $total_loops = shift;
    my $n = 1;
    print "Adding data to shared varable $total_loops times\n";

    until ($n > $total_loops) {
     if ($type_hash) {
      $share_hash{$n} = 'X' x 1111;
     }
     if ($type_scalar) {
      $share_var .= 'X' x 1111;
     }
     $n += 1;
    }
    print "Done Adding Data\n";
}

#### Clean up Data
sub Clean_Data {
    print "Clean Up Data\n";

    if ($type_hash) {
     ## Method to fix hash (Trying Everything i can think of!
     my $n = 1;
     my $total_loops = 100_000;
     until ($n > $total_loops) {
      undef $share_hash{$n};
      $n += 1;
     }

     %share_hash = ();
     $share_hash = ();
     undef $share_hash;
     undef %share_hash;
    }
    if ($type_scalar) {
     undef $share_var;
    }
}

#### Check Memory Usage
sub Memory_Check {
    ## Get current memory from shell
    my @mem = `ps aux | grep \"$$\"`;
    my($results) = grep !/grep/, @mem;

    ## Parse Data from Shell
    chomp $results;
    $results =~ s/^\w*\s*\d*\s*\d*\.\d*\s*\d*\.\d*\s*//g; $results =~ s/pts.*//g;
    my ($vsz,$rss) = split(/\s+/,$results);

    ## Format Numbers to Human Readable
    my $h = Number::Bytes::Human->new();
    my $virt = $h->format($vsz);
    my $h = Number::Bytes::Human->new();
    my $res = $h->format($rss);

    print "Current Memory Usage: Virt: $virt  RES: $res\n";

    if ($type_hash) {
     my $total_size = total_size(\%share_hash);
     my @arr_c = keys %share_hash;
     print "Length of Hash: " . ($#arr_c + 1) . "  Hash Mem Total Size: $total_size\n";
    }
    if ($type_scalar) {
     my $total_size = total_size($share_var);
     print "Length of Scalar: " . length($share_var) . "  Scalar Mem Total Size: $total_size\n";
    }

}

OUTPUT:

./Memory_Undef_Simple.cgi 
Current Memory Usage: Virt: 6.9K  RES: 2.7K
Length of Hash: 0  Hash Mem Total Size: 92
Length of Scalar: 0  Scalar Mem Total Size: 12
Hit Enter (add to memory): 
Adding data to shared varable 100000 times
Done Adding Data
Current Memory Usage: Virt: 228K  RES: 224K
Length of Hash: 100000  Hash Mem Total Size: 116813243
Length of Scalar: 111100000  Scalar Mem Total Size: 111100028
Hit Enter (Set Varable to nothing): 
Current Memory Usage: Virt: 228K  RES: 224K
Length of Hash: 100000  Hash Mem Total Size: 116813243
Length of Scalar: 0  Scalar Mem Total Size: 111100028
Hit Enter (clean data): 
Clean Up Data
Current Memory Usage: Virt: 139K  RES: 135K
Length of Hash: 0  Hash Mem Total Size: 92
Length of Scalar: 0  Scalar Mem Total Size: 24
Hit Enter (start over): 

So as you can see the memory goes down, but it only goes down the size of the scalar. Any ideas how to free the memory of the hash?

Also Devel::Size shows the hash is only taking up 92 bytes even though the program still is using 139K.

Answer 1

In general, you cannot expect perl to release memory to the OS.

See the FAQ: How can I free an array or hash so my program shrinks?

It is always a good idea to read the FAQ list, also installed on your computer, before wasting your time.

Answer 2

Generally, yeah, that's how memory management on UNIX works. If you are using Linux with a recent glibc, and are using that malloc, you can return free'd memory to the OS. I am not sure Perl does this, though.

If you want to work with large datasets, don't load the whole thing into memory, use something like BerkeleyDB:

http://search.cpan.org/dist/BerkeleyDB/BerkeleyDB.pod

Example code, stolen verbatim:

  use strict ;
  use BerkeleyDB ;

  my $filename = "fruit" ;
  unlink $filename ;
  tie my %h, "BerkeleyDB::Hash",
              -Filename => $filename,
              -Flags    => DB_CREATE
      or die "Cannot open file $filename: $! $BerkeleyDB::Error\n" ;

  # Add a few key/value pairs to the file
  $h{apple}  = "red" ;
  $h{orange} = "orange" ;
  $h{banana} = "yellow" ;
  $h{tomato} = "red" ;

  # Check for existence of a key
  print "Banana Exists\n\n" if $h{banana} ;

  # Delete a key/value pair.
  delete $h{apple} ;

  # print the contents of the file
  while (my ($k, $v) = each %h)
    { print "$k -> $v\n" }

  untie %h ;

(OK, not verbatim. Their use of use vars is ... legacy ...)

You can store gigabytes of data in a hash this way, and you will only use a tiny bit of memory. (Basically, whatever BDB's pager decides to keep in memory; this is controllable.)

Answer 3

Why do you want Perl to release the memory to the OS? You could just use a larger swap.

If you really must, do your work in a forked process, then exit.

Answer 4

Try recompiling perl with the option -Uusemymalloc to use the system malloc and free. You might see some different results