Why don't my variables interpolate correctly into my Perl substitution pattern?

Question

I'm writing a complex script that takes the XML backup of a Blogger blog and converts it to InDesign Tagged Text to be laid out in a book. I'm using a whole bunch of regular expressions to clean out the HTML tags of each blog post and convert them to InDesign tags. For example:

<p>A really long paragraph.</p> -> <ParaStyle:Main text>A really long paragraph.
<em>Whatever</em> -> <CharStyle:Italic>Whatever<CharStyle:>

For the most part the script is working great. However, InDesign can't handle nested tags. <CharStyle:Small><CharStyle:Italic>This is small italic text<CharStyle:><CharStyle:> will not work and needs to end up as <CharStyle:Small italic>This is small italic text<CharStyle:>

I'm trying to use variables in regex search patterns to find anywhere where character style tags are doubled up, but when I use the variables, nothing is found. If I hard code the InDesign tags into the regex, though, it works. What is making the variables unfindable?

Here's a working excerpt from my code (in real life $input isn't a string variable, but a LibXML object that the script parses...this is just for illustration)

#!/usr/bin/perl -w
use strict;

my $IDitalic = "<~~CharStyle:Italic>";
my $IDsmall = "<~~CharStyle:Small>";
my $IDsmallitalic = "<~~CharStyle:Small italic>";
my $IDcharend = "<~~CharStyle:>";

sub cleanText {
    my $text = $_[0];

    # Replace any span with a font size attribute with "small" character style
    $text =~ s/<span[^>]*?font-size[^>]*>(.*?)<\/span>/$IDsmall$1$IDcharend/gis;

    # Replace <em> tags with "italic" character style
    $text =~ s/<em>(.*?)<\/em>/$IDitalic$1$IDcharend/gis;

    #--------------------------------------------------------
    # Problem section
    #
    # The following works since everything is hard coded
    # $text =~ s/<~~CharStyle:Small><~~CharStyle:Italic>/$IDsmallitalic/gi;
    # $text =~ s/<~~CharStyle:><~~CharStyle:>/$IDcharend/gi;

    # When I use variables, though, it doesn't work...
    $text =~ s/{$IDsmall}{$IDitalic}/$IDsmallitalic/gi;
    $text =~ s/({$IDcharend})\1+/$1/gi;

    #--------------------------------------------------------


    # Clear out all tags that aren't the InDesign tags, take out the dummy ~~ and rebuild the actual tag
    $text =~ s/<[^~~](?:[^>'"]*|(['"]).*?\1)*>//gs;
    $text =~ s/<~~/</gs;

    return $text;
}

my $input = "<~~ParaStyle:Main text>In sodales malesuada nisi quis varius. Proin a ligula mauris. Proin ac justo est, vitae sollicitudin tortor. Proin auctor, <span style=\"font-size:78%\">augue eu</span> fringilla imperdiet, nisi sapien tempus libero, sed aliquet quam metus vel risus. Curabitur feugiat tristique porttitor. Integer malesuada volutpat accumsan. <span class=\"dummy\"In egestas</span> metus ut erat placerat tempus. <em>Nam vestibulum</em>, est quis scelerisque tincidunt, enim est lacinia ligula, vel accumsan ante nisl consectetur massa. Nullam velit nisi, viverra quis viverra ac, dictum ac enim. Sed nisl magna, fringilla at placerat quis, facilisis id nibh. Mauris eget sapien mauris, nec sollicitudin urna. Curabitur ac nunc a arcu vulputate tincidunt.\n<~~ParaStyle:Main text><span style=\"font-size:78%\"><em>**This is really small text</em></span>\n<ParaStyle:Comments\:Comment author>Andrew\n<~~ParaStyle:Comments\:Comment date>Friday, May 29, 2009— 8:15 PM";


print cleanText($input);

So, what's going wrong?

Also, is there a better way to maintain the InDesign tags without having the dummy tildes in the variable name?

Thanks!

Author apparently decided to parse the HTML, for more information go to the parsing follow-up question.

Answer 1

Try putting your '$' outside the '{'.. like so:

$text =~ s/${IDsmall}${IDitalic}/$IDsmallitalic/gi;
$text =~ s/(${IDcharend})\1+/$1/gi;

Answer 2

The first thing you are doing wrong is trying to use regexes on XML, as you have noticed, it doesn't work. This is a fundamental limitation of regexes. You should be using a parser instead. I like XML::Twig.

The second thing you are doing wrong is saying {$IDsmall} in the regex. That means a literal { the variable's contents then a literal }. Since the literal curly braces are in your text versions I assume you meant to type ${IDsmall}, however, that is unnecessary because the curly braces are only needed when you must make it clear what is a variable and what is text like this /${IDsmall}some other text/. In this case, without the curly braces Perl would think you were referring to a variable named $IDsmallsome.

The third thing you are doing wrong is not using \Q and \E to prevent special characters in your variables from affecting the match: /\Q$IDsmall\E/. Of course, if you meant for the special characters to affect the match, then you shouldn't be using a normal string. You should be using a quoted regex made by the qr// operator.

The fourth thing you are doing wrong is trying to use a negated character class to match more than one character: <[^~~](?:[^>'"]*|(['"]).*?\1)*>. /[^~~]/ means the same thing as /[^~]/. You probably want /[^~]{2}/.

There may be other problems, those were just what I saw on a first glance.