How can I parse incomplete XML fragments with Perl's XML::Twig?

Question

I'm trying to extract data from log files in XML format. As these are huge, I am using XML::Twig to extract the relevant data from a buffer instead of the whole file(s)

As these are concatenaded data from STDIN, the XML is far from well formed. So frequently the parser stops with an error. How can I get the XML parser to ignore the errors and only extract the tags I am interested in? Do I have to fall back to regular expression parsing (start-tag - end-tag)?

Answer 1

I would actually just accumulate the data between <message></message> tags and then parse that string, assuming the content of each message is small:

#!/usr/bin/perl

use strict; use warnings;

use XML::Simple;
use Data::Dumper;

my $in_message;
my $message;

LOGENTRY:
while ( my $line = <DATA> ) {
    while ( $line =~ /^<message/ .. $line =~ m{</message>$} ) {
        $message .= $line;
        next LOGENTRY;
    }
    if ( $message ) {
        process_message($message);
        $message = '';
    }
}

sub process_message {
    my ($message) = @_;

    my $xml = XMLin(
        $message,
        ForceArray => 1,
    );
    print Dumper $xml;
}

__DATA__
ldksj
lskdfj
lksd

sdfk

<message sender="1">Hi</message>

sdk
dkj

<message sender="2">Hi yourself!</message>

sd

Output:

$VAR1 = {
          'sender' => '1',
          'content' => 'Hi'
        };
$VAR1 = {
          'sender' => '2',
          'content' => 'Hi yourself!'
        };