Why does my 'use my_module;' take so much heap memory?

Question

This sample script:

#!/usr/bin/perl -w

while (1)
{
  sleep(1);
}

takes about 264 kB

grep -A1 heap  /proc/9216/smaps 
0817b000-081bd000 rw-p 0817b000 00:00 0          [heap]
Size:               264 kB

but when I only add my module:

#!/usr/bin/perl -w

use my_module;

while (1)
{
  sleep(1);
}

it takes 18092 kB !

grep -A1 heap  /proc/9219maps 
0817b000-09326000 rw-p 0817b000 00:00 0          [heap]
Size:             18092 kB

Note: The 'my_module' has a lot of 'use module;' inside it.

How can I find what takes so much memory ?

How can I reduce it ? (using 'use module (function)' ?)

Thanks for your help.

Answer 1

You didn't leave any specifics, but you already answered the general question: Yes, using lots of modules can consume a lot of memory, depending on what those modules are and what they import. Absolutely, you will use less memory if you use fewer modules, or selectively import symbols (or none at all).

Can you provide a more specific example of something that is consuming more memory than you think it should?

Answer 2

To answer: How can I reduce it ? (using 'use module (function)' ?)

Instead of a large amount of "use", perhaps just using "require" when you need to use each module within your "my_module". That's what I do to cut back on time/size. That is if you don't need every single module that you're "use"ing in your "my_module".

Answer 3

Insert BEGIN {} blocks to narrow down the culprit as in

#! /usr/bin/perl

sub grep_heap { print @_, "\n"; system "grep -A1 heap /proc/$$/smaps" }

BEGIN { grep_heap "<null>" }
use warnings;
BEGIN { grep_heap "+warnings" }
use strict;
BEGIN { grep_heap "+strict" }

use Data::Dumper;
BEGIN { grep_heap "+Data::Dumper" }
use POSIX;
BEGIN { grep_heap "+POSIX" }

print "Hi\n";

On my Linux host, I see

$ ./prog.pl
<null>
0889b000-088de000 rw-p 0889b000 00:00 0                                  [heap]
Size:               268 kB
+warnings
0889b000-08920000 rw-p 0889b000 00:00 0                                  [heap]
Size:               532 kB
+strict
0889b000-08920000 rw-p 0889b000 00:00 0                                  [heap]
Size:               532 kB
+Data::Dumper
0889b000-089a4000 rw-p 0889b000 00:00 0                                  [heap]
Size:              1060 kB
+POSIX
0889b000-08ace000 rw-p 0889b000 00:00 0                                  [heap]
Size:              2252 kB
Hi

As for what to do about it, you could implement replacement modules with scaled down functionality or ask yourself whether you really need a particular module at all. However, in general, perl's design prefers to throw memory at problems, and these days it's common for machines to have multiple gibibytes of main memory.

Is this resource issue causing performance problems?

Below is a program that reads through the list of modules in perlmodlib.pod and for each module forks a child to require and import it and check its own heap.

#! /usr/bin/perl

sub heap {
  my($heap) = @_;
  unless ($heap =~ /^([0-9a-f]+)-([0-9a-f]+)/m) {
    warn "$0: unexpected heap:\n$heap";
    return -1;
  }
  hex($2) - hex($1);
}

sub size {
  my($bytes) = @_;

  my @units = (
    [ MiB   => "%.1f", 1_024 * 1_024 ],
    [ KiB   => "%.1f", 1_024 ],
  );

  for (@units) {
    my($unit,$fmt,$n) = @$_;
    return sprintf "$fmt %s" => $bytes/$n, $unit
      if $bytes >= $n;
  }

  return "$bytes byte" . ($bytes == 1 ? "" : "s");
}

my %incr;

my $perlmodlib = `perldoc -l perlmodlib`;
die "$0: perldoc failed" unless defined $perlmodlib;

my $base = heap `grep heap /proc/$$/smaps`;
warn "$0: base=" . size($base) . "\n";

chomp $perlmodlib;
open my $fh, "<", $perlmodlib   or die "$0: open $perlmodlib: $!";

while (<$fh>) {
  next unless /^=head2 Pragmatic Modules/ ..
              /^=head2 Extension Modules/;

  if (/^=item (\w+(::\w+)*)/) {
    my $mod = $1;
    (my $path = "$mod.pm") =~ s!::!/!g;

    my $pid = open my $fh, "-|";
    die "$0: fork: $!" unless defined $pid;

    if ($pid == 0) {
      open STDERR, ">", "/dev/null" or warn "$0: open: $!";
      exec "perl", "-e", <<EOProgram;
BEGIN {
  require \"$path\";
  eval { $mod->import };
  system qq(grep heap /proc/\$\$/smaps);
}
EOProgram
      die "$0: exec: $!";
    }
    else {
      local $/;
      my $heap = <$fh>;
      unless (defined $heap && length $heap) {
        warn "$0: use $mod failed";
        next;
      }
      $heap = heap $heap;
      $incr{$mod} = $heap > 0 ? $heap-$base : $heap;
    }
  }
}

foreach my $mod (sort { $incr{$b} <=> $incr{$a} } keys %incr) {
  print "$mod - ", size($incr{$mod}), "\n";
}

The top few offenders for perl-5.8.8:

CPAN::Nox - 9.7 MiB
CPAN - 9.6 MiB
ExtUtils::MM_VMS - 5.3 MiB
CPAN::FirstTime - 5.2 MiB
ExtUtils::Installed - 5.2 MiB
B::CC - 5.2 MiB
bigrat - 4.9 MiB
Math::BigRat - 4.8 MiB
ExtUtils::MM_NW5 - 4.7 MiB
ExtUtils::MM_OS2 - 4.6 MiB
ExtUtils::MM_Win32 - 4.6 MiB
ExtUtils::MM_Win95 - 4.6 MiB