ansaurus

Question

perl how to regex parts of data instead of entire string and then print out a csv file

Answer 1

+4 A:

Use 'capturing parentheses' to identify the bits you want to deal with.

   if ($line =~ m/(.* $mon $day) \d{2}:\d{2}:\d{2} $year:.*
                  (ERROR:|backup-date=|backup-size=|
                   backup-time=|backup-status)/x) {

You will need to do some surgery on the second set of parentheses - those surrounding the start of the various keywords. You may have to chop those out in bits and pieces inside the condition.

When you have all the data extracted into variables, use Text::CSV to handle CSV output (and input).

There are a myriad modules to handle HTML or XML (over 2000, and I think over 3000, with HTML in their name - I happened to look yesterday). Many of those won't be applicable, but CPAN is your friend.

Answering questions posed by comments

Would I split them off into separate variables as well? The first part gives me the date/time that I need. The next filter then gives me 1) Error: 2)backup-date= 3)backup-size= ...etc.

More or less. Unfortunately, you don't show some representative input lines, which means it is hard to tell what might be best. However, it seems likely that a scheme such as:

while (my $line = <$fh>)
{
    chomp $line;
    if ($line =~ m/(.* $mon $day) \d\d:\d\d:\d\d $year:/)
    {
        my $date = $1;
        my %items = ();
        $line =~ s/.* $mon $day \d\d:\d\d:\d\d $year://;
        while ($line =~ m/(ERROR|backup-date|backup-size|
                           backup-time|backup-status)
                          [:=]([^:]+)/x)
        {
            my $key = $1;
            my $val = $2;
            $items{$key} = $val;
            $line =~ s/$key[:=]$val[:=]?//;
        }
        # The %items hash contains the split out information.
        # Now write the data for this line of the log file.
    }
}

There might well be better ways to handle the trimming (but it is Perl so TMTOWTDI), but the basic idea here is to catch the lines that are interesting, then progressively chop the bits of interest out of the line, so the line grows shorter on each iteration (therefore, eventually terminating the inner while loop).

Note the use of the /x modifier to allow for a more readable regex split over lines (I edited the original answer version to use that too). I've also allowed 'ERROR' to be followed by an '=' or the other keywords to be followed by ':'; it seems unlikely that you'd get false matches that way, and it simplifies the regex substitute operations. The initial pattern match no longer requires one of the subsections to be present, either. You must judge for yourself whether those small changes (which might pick up non-conforming information) matter or not. For most of my purposes, the chance of the mismatch is small enough not to be an issue - but for legal reasons, it might not be acceptable to you.

Answering questions posed by 'answer'

I manufactured some data:

Wed Jul 30 00:49:51 2010: test203.bak_lvm:backup:INFO: backup-size=417.32 GB
Wed Jul 30 00:49:52 2010: test203.bak_lvm:backup:INFO: backup-time=04:49:51
Wed Jul 30 00:49:53 2010: test203.bak_lvm:backup:INFO: backup-status=Backup succeeded
Wed Jul 30 00:49:51 2010: backup-size=417.32 GB:backup-time=04:49:51:backup-status=Backup succeeded

I took the script in the answer and hacked and instrumented it - making it standalone. I also removed the dependency on specific files - it reads standard input and writes to standard output. It makes my testing easier - and the code more flexible.

use strict;
use warnings;
use constant debug => 0;
my $mon = 'Jul';
my $day = 30;
my $year = 2010;

while (my $line = <>)
{
    chomp $line;
    print "Line: $line\n" if debug;
    if ($line =~ m/(.* $mon $day) \d\d:\d\d:\d\d $year:/) #Mon Jul 26 22:00:02 2010:
    {
        print "### Scan\n";
        my $date = $1;
        print "$date\n";
        my %items = ();
        $line =~ s/.* $mon $day \d\d:\d\d:\d\d $year://;
        print "Line: $line\n" if debug;
        while ($line =~ m/(ERROR|backup-date|backup-size|backup-time|backup-status)[:=]([^:]+)/)
        {
            my $key = $1;
            my $val = $2;
            $items{$key} = $val;
            $line =~ s/$key[:=]$val[:=]?//;
            print "$key=$val\n";
            print "Line: $line\n" if debug;
        }
        print "### Verify\n";
        for my $key (sort keys %items)
        {
            print "$key = $items{$key}\n";
        }
    }
}

The output I get is:

### Scan
Wed Jul 30
backup-size=417.32 GB
### Verify
backup-size = 417.32 GB
### Scan
Wed Jul 30
backup-time=04
### Verify
backup-time = 04
### Scan
Wed Jul 30
backup-status=Backup succeeded
### Verify
backup-status = Backup succeeded
### Scan
Wed Jul 30
backup-size=417.32 GB
backup-time=04
backup-status=Backup succeeded
### Verify
backup-size = 417.32 GB
backup-status = Backup succeeded
backup-time = 04

The verify loop prints out the data from the '%items' hash quite happily. With the debug value set to 1 instead of 0, the output I get is:

Line: Wed Jul 30 00:49:51 2010: test203.bak_lvm:backup:INFO: backup-size=417.32 GB
### Scan
Wed Jul 30
Line:  test203.bak_lvm:backup:INFO: backup-size=417.32 GB
backup-size=417.32 GB
Line:  test203.bak_lvm:backup:INFO: 
### Verify
backup-size = 417.32 GB
Line: Wed Jul 30 00:49:52 2010: test203.bak_lvm:backup:INFO: backup-time=04:49:51
### Scan
Wed Jul 30
Line:  test203.bak_lvm:backup:INFO: backup-time=04:49:51
backup-time=04
Line:  test203.bak_lvm:backup:INFO: 49:51
### Verify
backup-time = 04
Line: Wed Jul 30 00:49:53 2010: test203.bak_lvm:backup:INFO: backup-status=Backup succeeded
### Scan
Wed Jul 30
Line:  test203.bak_lvm:backup:INFO: backup-status=Backup succeeded
backup-status=Backup succeeded
Line:  test203.bak_lvm:backup:INFO: 
### Verify
backup-status = Backup succeeded
Line: Wed Jul 30 00:49:51 2010: backup-size=417.32 GB:backup-time=04:49:51:backup-status=Backup succeeded
### Scan
Wed Jul 30
Line:  backup-size=417.32 GB:backup-time=04:49:51:backup-status=Backup succeeded
backup-size=417.32 GB
Line:  backup-time=04:49:51:backup-status=Backup succeeded
backup-time=04
Line:  49:51:backup-status=Backup succeeded
backup-status=Backup succeeded
Line:  49:51:
### Verify
backup-size = 417.32 GB
backup-status = Backup succeeded
backup-time = 04

The substitute operations delete the previously matched part of the line. There are ways of continuing a match where you left off - see \G at the 'perlre' page.

Note that the regex is crafted to stop at the first colon after the 'colon or equals' after the keyword. That means it truncates the backup time. One moral is "do not use a separator that can appear in the data". Another is "provide sample data so people can help you more easily". Another is "provide complete but minimal working scripts where possible".

Processing the sample data

Now that we have the sample input data, we can see that you need slightly different processing. This script:

use strict;
use warnings;
use constant debug => 0;
my $mon = 'Jul';
my $day = 28;
my $year = 2010;
my %items = ();

while (my $line = <>)
{
    chomp $line;
    print "Line: $line\n" if debug;
    if ($line =~ m/(.* $mon $day) \d\d:\d\d:\d\d $year: ([^:]+):backup:/) #Mon Jul 26 22:00:02 2010:
    {
        print "### Scan\n" if debug;
        my $date = $1;
        my $set = $2;
        print "$date ($set): " if debug;
        $items{$set}->{'a-logdate'} = $date;
        $items{$set}->{'a-dataset'} = $set;
        if ($line =~ m/(ERROR|backup-date|backup-size|backup-time|backup-status)[:=](.+)/)
        {
            my $key = $1;
            my $val = $2;
            $items{$set}->{$key} = $val;
            print "$key=$val\n" if debug;
        }
    }
}

print "### Verify\n";
for my $set (sort keys %items)
{
    print "Set: $set\n";
    my %info = %{$items{$set}};
    for my $key (sort keys %info)
    {
        printf "%s=%s;", $key, $info{$key};
    }
    print "\n";
}

produces this result on the sample data file.

### Verify
Set: db9.abc.bak
a-dataset=db9.abc.bak;a-logdate=Wed Jul 28;backup-date=20100728000004;
Set: test203.bak_lvm
a-dataset=test203.bak_lvm;a-logdate=Wed Jul 28;backup-size=417.32 GB;backup-status=Backup succeeded;backup-time=04:49:51;

Note that now we have sample data, we can see that there is only one key/value pair per line, but there are multiple systems backed up per day. So, the inner while loop becomes a simple if. The printing out occurs at the end. And I'm using a 'two-tier' hash. The %items contains an entry for each data set; the entry, though, is a reference to a hash. Not necessarily something for novices to play with, but it fell into place very naturally with the previous code. Note, too, that this version doesn't hack the line - there's no need since there's only one lot of data per line.

Can it be improved - yes, undoubtedly. Does it work? Yes, more or less... Can it be hacked into shape? Yes, it can be hacked to work as you need.

Jonathan Leffler 2010-07-29 22:47:15

Would i split them off into separate variables as well? The first part gives me the date/time that I need, next filter then is give me 1) Error: 2)backup-date= 3)backup-size= ...etc.

jda6one9 2010-07-29 23:11:41

Thank you Johnathan! This helps a great deal. I put out a couple of print statements to understand this more. Thanks again.

jda6one9 2010-07-30 16:19:34

@Johnathan - Can you elaborate more on what you mean by "The initial pattern match no longer requires one of the subsections to be present, either."? Are you referring to "if ($line =~ m/(.* $mon $day) \d\d:\d\d:\d\d $year:/)"? Its also shown 4 lines down..Thanks.

jda6one9 2010-07-30 17:03:04

@jaedre619: yes - the 'if' line does not insist that one of 'ERROR' or 'backup-date' or ... is present in what it matches. You will have to decide whether that matters. It would matter if there are lines in the log file which match the date part but contain none of the other magic sections; it won't matter if there are no such lines.

Jonathan Leffler 2010-07-30 17:11:14

@Jonathan - If you can also pls elaborate also on the following. I'm trying to understand the lines where you use-> $line =~ s/.* $mon $day \d\d:\d\d:\d\d $year://; AND $line =~ s/$key[:=]$val[:=]?//; I read it as after searching for the match, replace with blank. What is the purpose of that? Is this emptying the hash to store the current value? So the hash is really storing one value? Sorry for all the questions, just trying to understand this more. thanks again.

jda6one9 2010-07-31 05:15:36

@jaeddre619: it is actually deleting the matched information. Try printing out $line after each iteration; you should see stuff disappear.

Jonathan Leffler 2010-07-31 05:43:04

Answer 2

A:

@Jonathan- I wrote out the text file within the while loop. It seems to work. I tried doing it after the second while loop as you suggested in your comment. I'm not sure why it didn't work.

open (my $MYDATAFILE, ">/home/test/myout.txt") || die "cannot append $!";
open(my $fh,"ssh $server cat $file |") or die "can't open log $server:$file: $!\n";
while (my $line = <$fh>)
{
    chomp $line;
    if ($line =~ m/(.* $mon $day) \d\d:\d\d:\d\d $year:/) #Mon Jul 26 22:00:02 2010:
    {
        my $date = $1;
        #print $date;
        my %items = ();
        $line =~ s/.* $mon $day \d\d:\d\d:\d\d $year://;
        while ($line =~ m/(ERROR|backup-date|backup-size|backup-time|backup-status)[:=]([^:]+)/)
        {
            my $key = $1;
            my $val = $2;
            $items{$key} = $val;
            $line =~ s/$key[:=]$val[:=]?//;
            #print "[$key]";
            #print "[$val]";
            print $MYDATAFILE "$key=$val";
        }
        # The %items hash contains the split out information.
        # Now write the data for this line of the log file.
    }
}

jda6one9 2010-07-30 19:34:44

@jaedre619: please edit the question to show a couple of sample lines of input, preferably the ones corresponding to your sample output (as now).

Jonathan Leffler 2010-07-30 22:43:10

Also, as a general guideline, it is better to edit your question and add material like this - maybe after a horizontal rule '`<hr>`' and with a title '`### Additional information`'. There are those who down-vote 'answers' which are actually amendments to the question.

Jonathan Leffler 2010-07-30 22:45:27

All suggestions noted. I just added a better sample of my raw logfile to avoid confusion of what I need. Thanks for the input.

jda6one9 2010-07-31 01:36:40

ansaurus

tags:

views:

answers:

perl how to regex parts of data instead of entire string and then print out a csv file

Answering questions posed by comments

Answering questions posed by 'answer'

Processing the sample data

related questions