How to find the amount of physical memory occupied by a hash in Perl?

Question

I have a Perl script where I maintain a very simple cache using a hash table. I would like to clear the hash once it occupies more than n bytes, to avoid Perl (32-bit) running out of memory and crashing.

I can do a check on the number of keys-value pairs:

if (scalar keys %cache > $maxSize)
 {
 %cache = ();
 }

But is it possible to check the actual memory occupied by the hash?

Answer 1

You're looking for Devel::Size

NAME

Devel::Size - Perl extension for finding the memory usage of Perl variables

SYNOPSIS

use Devel::Size qw(size total_size);

my $size = size("A string");
my @foo = (1, 2, 3, 4, 5);
my $other_size = size(\@foo);
my $foo = {a => [1, 2, 3],
        b => {a => [1, 3, 4]}
       };
my $total_size = total_size($foo);

Answer 2

You can use Devel::Size to determine the memory used, but you can't generally give return memory to the OS. It sounds like you're just trying to clear and reuse, though, which should work fine.

If the cache is for a function, consider using the Memoize module instead of maintaining the cache yourself. It supports cache expiration (via Memoize::Expire) so you can limit the size of the cache without destroying it entirely.

Answer 3

You can install Devel::Size to find out the memory taken by any construct in Perl. However do be aware that it will take a large amount of intermediate memory, so I would not use it against a large data structure. I would certainly not do it if you think you may be about to run out of memory.

BTW there are a number of good modules on CPAN to do caching in memory and otherwise. Rather than roll your own I would suggest using one of them instead. For instance try Tie::Cache::LRU for an in-memory cache that will only go up to a specified number of keys.

Answer 4

Devel::Size is the answer to your question. (Note that Devel::Size will temporarily allocate a significant amount of memory when processing a large data structure, so it's not really well suited to this purpose.)

However, Cache::SizeAwareMemoryCache and Tie::Cache already implement what you're looking for (with somewhat different interfaces), and could save you from reinventing the wheel.

Memoize is a module that makes it simple to cache the return value from a function. It doesn't implement a size-based cache limit, but it should be possible to use Tie::Cache as a backend for Memoize.

Answer 5

Cache::Memory

use Cache::Memory;

my $cache = Cache::Memory->new(
  namespace => 'MyNamespace',
  default_expires => '600 sec'
);

my $size  = $cache->size()
my $limit = $cache->size_limit();

Answer 6

If you're worrying about managing the amount of memory that Perl is using, you should probably look at an alternative approach. Why do you need that much in RAM all at once? Should you be using some sort of persistence system?

Answer 7

As others have said, caching is not a wheel you need to re-invent, there's plenty of simple caching solutions on CPAN which will do the job nicely for you.

Cache::SizeAwareMemoryCache can be told the maximum size you want it to use, then you can leave it to care about the cache for you.