Which KiokuDB backend is suitable for my serialization needs?

Question

I use KiokuDB to store a couple of Moose objects and a couple of simple array structures (hashes and arrays).

I do not need any fancy searches, transactions etc., simple the ability to fetch (lookup) an object. Also, as soon as I'm done creating the DB, it can be set read-only. No changes will ever be made to it.

The main (only?) reason I use KiokuDB is to keep object graph.

The largest object, which dominants the total size of the DB, is a Moose object that has a relatively large array in it (let's call this object large_obj). Previously, I stored large_obj (alone) using Storable + PerlIO::gzip or even JSON + PerlIO::gzip. It worked fine and I was very happy with the results (the use of gzip compressed the store file to about 5% of its original size).

There is another, smaller Moose object, which is basically an array of some 20-30k small Moose objects.

Now, after migrating to KiokuDB, I first used the simple Hash backend, then dumped it to a file (using Cmd) with PerlIO::gzip again. This worked very well in cases where large_obj was relatively small, but once it got larger, I just got out of memory errors. I guess the hash backed is not suitable for large objects.

I then tried the recommended Berkeley backend, although it seems like an overkill (as mentioned, I don't really need all the fancy DB capabilities). It works much slower than the original Storable + PerlIO::gzip solution, it takes far more space, and it also runs out of memory for larger objects! (I use a 3GB RAM ubuntu).

I also tried the Files backend, but it fails with:

Too many open files at /usr/local/perls/perl-5.12.2/lib/site_perl/5.12.2/Directory/Transactional.pm line 130.
    (in cleanup) Too many open files at /usr/local/perls/perl-5.12.2/lib/site_perl/5.12.2/Directory/Transactional.pm line 130.

Do you have any suggestions on how can I store my objects in a way that is both space-efficient and maintains the object graph?

Answer 1

Implement your own backend using Data::Serializer:

package KiokuDB::Backend::Serialize::Data::Serializer;
use Moose;
use Moose::Role;

use Data::Serializer;

use namespace::clean -except => 'meta';

with qw(
    KiokuDB::Backend::Serialize
    KiokuDB::Backend::Role::UnicodeSafe
    KiokuDB::Backend::Role::BinarySafe
);

has '_serializer' => (
    is       => 'ro',
    isa      => 'Data::Serializer',
    required => 1,
    lazy     => 1,
    default  => sub {
        Data::Serializer->new(
            serializer => 'FreezeThaw', # Storable, FreezeThaw, Data::Denter, Config::General, YAML, PHP::Serialization, XML::Dumper, and Data::Dumper
            digester   => 'MD5', # See http://search.cpan.org/~gaas/Digest-1.16/Digest.pm#Digest_speed
            compress   => 1,
            compressor => 'Compress::Zlib', # Compress::Zlib or Compress::PPMd
        );
    },
);

sub serialize {
    my ( $self, $entry ) = @_;

    return $self->_serializer->serialize($entry);
}

sub deserialize {
    my ( $self, $blob ) = @_;

    return $self->_serializer->deserialize($blob);
}

sub serialize_to_stream {
    my ( $self, $fh, $entry ) = @_;

    $self->_serializer->store( $entry, $fh );
}

sub deserialize_from_stream {
    my ( $self, $fh ) = @_;

    $self->_serializer->retrieve($fh);
}

__PACKAGE__