How do I extract lines between two line delimiters in Perl?

Question

I have an ASCII log file with some content I would like to extract. I've never taken time to learn Perl properly, but I figure this is a good tool for this task.

The file is structured like this:

... 
... some garbage 
... 
... garbage START
what i want is 
on different
lines 
END 
... 
... more garbage ...
next one START 
more stuff I want, again
spread 
through 
multiple lines 
END 
...
more garbage

So, I'm looking for a way to extract the lines between each "START" and "END" delimiter strings. How can I do this?

So far, I've only found some examples on how to print a line with the "START" string, or other documentation items that are somewhat related with what I'm looking for.

Answer 1

http://stackoverflow.com/questions/1040657/how-can-i-grab-multiple-lines-after-a-matching-line-in-perl

How's that one? In that one, the END string is $^, you can change it to your END string.

I am also a novice, but the solutions there provide quite a few methods... let me know more specifically what it is you want that differs from the above link.

Answer 2

You want the flip-flop operator (better known as the range operator) ..

#!/usr/bin/env perl
use strict;
use warnings;

while (<>) {
  if (/START/../END/) {
    next if /START/ || /END/;
    print;
  }
}

Replace the call to print with whatever you actually want to do (e.g., push the line into an array, edit it, format it, whatever). I'm next-ing past the lines that actually have START or END, but you may not want that behavior. See this article for a discussion of this operator and other useful Perl special variables.

Answer 3

while (<>) {
    chomp;      # strip record separator
    if(/END/) { $f=0;}
    if (/START/) {
        s/.*START//g;
        $f=1;
    }
    print $_ ."\n" if $f;
}

try to write some code next time round

Answer 4

Dear all,

After Telemachus' reply, things started pouring out. This works as the solution I'm looking at after all.

1. I'm trying to extract lines delimited by two strings (one, with a line ending with "CINFILE="; other, with a line containing a single "#") in separate lines, excluding the delimiter lines. This I can do with Telemachus' solution.
2. The first line has a space I want to remove. I'm also including it.
3. I'm also trying to extract each line-set into separate files.

This works for me, although the code can be classified as ugly; this is because I'm currently a virtually newcomer to Perl. Anyway here goes:

#!/usr/bin/env perl
use strict;
use warnings;

my $start='CINFILE=$';
my $stop='^#$';
my $filename;
my $output;
my $counter=1;
my $found=0;

while (<>) {
  if (/$start/../$stop/) {
    $filename=sprintf("boletim_%06d.log",$counter);
    open($output,'>>'.$filename) or die $!;
    next if /$start/ || /$stop/;
    if($found == 0) { print $output (split(/ /))[1]; }
    else { print $output $_; }
    $found=1;
  } else { if($found == 1) { close($output); $counter++; $found=0; } }
}

I hope it benefits others as well. Cheers.

Answer 5

From perlfaq6's answer to How can I pull out lines between two patterns that are themselves on different lines?

---

You can use Perl's somewhat exotic .. operator (documented in perlop):

perl -ne 'print if /START/ .. /END/' file1 file2 ...

If you wanted text and not lines, you would use

perl -0777 -ne 'print "$1\n" while /START(.*?)END/gs' file1 file2 ...

But if you want nested occurrences of START through END, you'll run up against the problem described in the question in this section on matching balanced text.

Here's another example of using ..:

while (<>) {
 $in_header =   1  .. /^$/;
 $in_body   = /^$/ .. eof;
# now choose between them
} continue {
 $. = 0 if eof; # fix $.
}

Answer 6

Not too bad for coming from a "virtual newcommer". One thing you could do, is to put the "$found=1" inside of the "if($found == 0)" block so that you don't do that assignment every time between $start and $stop.

Another thing that is a bit ugly, in my opinion, is that you open the same filehandler each time you enter the $start/$stop-block.

This shows a way around that:

#!/usr/bin/perl

use strict;
use warnings;

my $start='CINFILE=$';
my $stop='^#$';
my $filename;
my $output;
my $counter=1;
my $found=0;

while (<>) {

    # Find block of lines to extract                                                           
    if( /$start/../$stop/ ) {

        # Start of block                                                                       
        if( /$start/ ) {
            $filename=sprintf("boletim_%06d.log",$counter);
            open($output,'>>'.$filename) or die $!;
        }
        # End of block                                                                         
        elsif ( /$end/ ) {
            close($output);
            $counter++;
            $found = 0;
        }
        # Middle of block                                                                      
        else{
            if($found == 0) {
                print $output (split(/ /))[1];
                $found=1;
            }
            else {
                print $output $_;
            }
        }

    }
    # Find block of lines to extract                                                           

}

Answer 7

How about if you know the START string but do not know the exact END string? Each START and END string begins with ">" so I am looking to find the ">START" string and print all the trailing lines until the next ">" is found.