How do I implement my own escape sequence when using split() in perl?

Question

I'm trying to write a parser for the EDI data format, which is just delimited text but where the delimiters are defined at the top of the file.

Essentially it's a bunch of splits() based on values I read at the top of my code. The problem is theres also a custom 'escape character' that indicates that I need to ignore the following delimiter.

For example assuming * is the delimiter and ? is the escape, I'm doing something like

use Data::Dumper;
my $delim = "*";
my $escape = "?";
my $edi = "foo*bar*baz*aster?*isk";

my @split = split("\\" . $delim, $edi);
print Dumper(\@split);

I need it to return "aster*isk" as the last element.

My original idea was to do something where I replace every instance of the escape character and the following character with some custom-mapped unprintable ascii sequence before I call my split() functions, then another regexp to switch them back to the right values.

That is doable but feels like a hack, and will get pretty ugly once I do it for all 5 different potential delimiters. Each delimiter is potentially a regexp special char as well, leading to a lot of escaping in my own regular expressions.

Is there any way to avoid this, possibly with a special regexp passed to my split() calls?

Answer 1

my @split = split( /(?<!\Q$escape\E)\Q$delim\E/, $edi);

will do the split for you, but you have to remove the escape characters separately:

s/\Q$escape$delim\E/$delim/g for @split;

Update: to allow the escape character to escape any character, including itself, not just the delimiter requires a different approach. Here's one way:

my @split = $edi =~ /(?:\Q$delim\E|^)((?:\Q$escape\E.|(?!\Q$delim\E).)*+)/gs;
s/\Q$escape$delim\E/$delim/g for @split;

*+ requires perl 5.10+. Before then, it would be:

/(?:\Q$delim\E|^)((?>(?:\Q$escape\E.|(?!\Q$delim\E).)*))/gs

Answer 2

Try Text::CSV.

Answer 3

This is a bit tricky if you want to handle the case where the escape character is the last character of a field correctly. Here's one way:

# Process escapes to hide the following character:
$edi =~ s/\Q$escape\E(.)/sprintf '%s%d%s', $escape, ord $1, $escape/esg;

my @split = split( /\Q$delim\E/, $edi);

# Convert escape sequences into the escaped character:
s/\Q$escape\E(\d+)\Q$escape\E/chr $1/eg for @split;

Note that this assumes that neither the escape char nor the delimiter will be a digit, but it does support the full range of Unicode characters.

Answer 4

Here's a custom function -- it's longer than ysth's answer, but in my opinion it's easier to break down into useful pieces (not being all one regex), and it also has the ability to cope with multiple delimiters that you asked for.

sub split_edi {
  my ($in, %args) = @_;
  die q/Usage: split_edi($input, escape => "#", delims => [ ... ]) /
    unless defined $in and defined $args{escape} and defined $args{delims};

  my $escape = quotemeta $args{escape};
  my $delims = join '|', map quotemeta, @{ $args{delims} };

  my ($cur, @ret);

  while ($in !~ /\G\z/cg) {
    if ($in =~ /\G$escape(.)/mcg) {
      $cur .= $1;
    } elsif ($in =~ /\G(?:$delims)/cg) {
      push @ret, $cur; 
      $cur = '';
    } elsif ($in =~ /\G((?:(?!$delims|$escape).)+)/mcg) {
      $cur .= $1;
    } else {
      die "hobbs can't write parsers";
    }
  }
  push @ret, $cur if defined $cur;
  @ret;
}

The first line is argument parsing, backslashing the escape char as necessary, and building a regex fragment that matches any of the delimiters.

Then comes the matching loop:

• If we find the escape, skip over it and capture the following character as a literal bit of the output instead of treating it specially.
• If we find any of the delimiters, start a new record.
• Otherwise, capture characters until the next escape or delimiter.
• Stop when we reach end-of-string.

which is pretty straightforward and still has pretty solid performance. Like ysth's regex solutions, it's ratcheting -- it won't try to backtrack unnecessarily. Correctness isn't guaranteed if the escape or any of the delimiters is multi-character, although I actually think it's pretty much right :)

say for split_edi("foo*bar;baz*aster?*isk", delims => [qw(* ;)], escape => "?");
foo
bar
baz
aster*isk