Why am I getting an "Out of memory" error with Perl's XML::Simple?

Question

Edited::

Hi All,

I have a XML file like this,

        <message>
            <c1>
                <rrcConnectionSetupComplete>
                    <rrc-TransactionIdentifier>2</rrc-TransactionIdentifier>
                    <criticalExtensions>
                        <c1>
                            <rrcConnectionSetupComplete-r8>
                                <selectedPLMN-Identity> 1 </selectedPLMN-Identity>
                                <dedicatedInfoNAS> 07410109014290112345671000028020000f0 </dedicatedInfoNAS>
                            </rrcConnectionSetupComplete-r8>
                        </c1>
                    </criticalExtensions>
                </rrcConnectionSetupComplete>
            </c1>
        </message>

I am using Perl code like this to access the data in xml file (i should stick on to this format of accessing)

#!/usr/bin/perl

use strict;

use XML::Simple;

my $xml = new XML::Simple;

my $data = $xml->XMLin("uL-DCCH-Message.xml");

my $rrc_trans_identifier = $data->{'c1'}->{'rrcConnectionSetupComplete'}->{'rrc-TransactionIdentifier'};
print "rrc_trans_id :: $rrc_trans_identifier\n";

my $selected_plmn_id = $data->{c1}->{rrcConnectionSetupComplete}->{criticalExtensions}->{c1}->{'rrcConnectionSetupComplete-r8'}->{'selectedPLMN-Identity'};
print "plmn identity :: $selected_plmn_id\n";

my $rrc_dedicated_info_nas = $data->{c1}->{rrcConnectionSetupComplete}->{criticalExtensions}->{c1}->{'rrcConnectionSetupComplete-r8'}->{dedicatedInfoNAS};
print "dedicated info nas :: $rrc_dedicated_info_nas\n";

The output produced is,

rrc_trans_id :: 2
plmn identity ::  1
dedicated info nas ::  07410109014290112345671000028020000f0

Perl code using XML::Simple is working fine for smaller XML files (as shown in the above output).

But If XML file is large, then XML::Simple cannot handle and it is showing error message Ran out of memory.

My question is, Is there any other XML Parsers i can use, so that i can access the elements in XML file in the similar manner as shown above ???

If there is any other parsers available, can any one give an example by following the same conventions i am following for XML::Simple..

Kindly help me, i am struggling with this

Answer 1

Without being able to see the code or a representative example, I am reduced to guessing. My bet is that the XML file is very large and you are running out of memory trying to store the data structure. Switch to a stream style XML parser like XML::Twig.

Usually, huge XML files will be be a set of records and you wind up just looping over the data structure you build. A better, faster way of doing this is to process the file as you go:

#!/usr/bin/perl

use strict;
use warnings;

use XML::Twig;

sub process_record {
    my $rec = shift;
    print "@{$rec}{qw/field1 field2 field3/}\n";
}

my $t = XML::Twig->new(
    twig_handlers => {
        record => sub {
            my ($t, $elt) = @_;

            my %record = map {
                $_ => $elt->first_child($_)->text
            } qw/field1 field2 field3/;

            process_record(\%record);

            #get rid of any memory that is being held
            $t->purge;
        },
    }
);

$t->parse(\*DATA);

__DATA__
<root>
    <record>
        <field1>1</field1>
        <field2>a</field2>
        <field3>foo</field3>
    </record>
    <record>
        <field1>2</field1>
        <field2>b</field2>
        <field3>bar</field3>
    </record>
    <record>
        <field1>3</field1>
        <field2>c</field2>
        <field3>baz</field3>
    </record>
</root>

Answer 2

For large XML files you could use a SAX parser, like XML::SAX. See perldoc XML::SAX::Intro.
XML::Simple is better suited for parsing small files, e.g. configs.

Answer 3

Whenever XML::Simple isn't doing exactly what you want, it's time to use something better. It's designed to work with smallish hashes, not solve all of your XML needs. Besides that, you'll have to tell us a lot more about your situation before we can even guess at the problem.

When you run into a problem, try to reproduce it in a small test case. That might be as simple as a program to read the file and access it. Remember, you're the primary person responsible for solving your problems. :)

Answer 4

To give you the leg up here is your updated example in XML::Twig

use strict;
use warnings;
use XML::Twig;

XML::Twig->new(
    twig_handlers => {
        'c1/rrcConnectionSetupComplete' => \&my_stuff,
    }
)->parsefile( 'uL-DCCH-Message.xml' );

sub my_stuff {
    my ($twig, $elt) = @_;

    my $rrc_trans_identifier = $elt->field( 'rrc-TransactionIdentifier' );
    print "rrc_trans_id :: $rrc_trans_identifier\n";

    my $twiglet = $elt->first_child('criticalExtensions')
                      ->first_child('c1')
                      ->first_child('rrcConnectionSetupComplete-r8');


    my $selected_plmn_id = $twiglet->first_child_text('selectedPLMN-Identity');
    print "plmn identity :: $selected_plmn_id\n";    

    my $rrc_dedicated_info_nas = $twiglet->first_child_text('dedicatedInfoNAS');
    print "dedicated info nas :: $rrc_dedicated_info_nas\n";

    $twig->purge;
}

This outputs exactly what you were after:

rrc_trans_id :: 2
plmn identity ::  1 
dedicated info nas ::  07410109014290112345671000028020000f0 

Important

• Above works for the example you provided but there are a few different ways you could achieve the same result in XML::Twig depending on what your exact requirements are.

• Please don't assign me with the accepted answer for this question! Chas & eugene were first on the scene with the correct answer to your problem and brian followed up with the ubiquitous answer.

/I3az/