Why do Perl string operations on Unicode characters add garbage to the string?

Question

A quick question for a change.

Perl:

$string =~ s/[áàâã]/a/gi; #This line always prepends an "a"
$string =~ s/[éèêë]/e/gi;
$string =~ s/[úùûü]/u/gi;

This Regex should convert "été" into "ete". What it does instead is converting it to "aetae". In other words, it prepends an "a" to every matched element. Even "à" is converted to "aa".

If I change the first line to this

$string =~ s/(á|à|â|ã)/a/gi;

it works, but... now it prepends an e to every matched element (like "eetee").

Even though I found a suitable solution, I am still curious as to why it behaves that way. Any ideas?

Edit 1:

I added "use utf8;" but it did not change the behavior (although it broke my output in Javascript/AJAX).

Edit2:

The Stream originates from an AJAX Request, performed by jQuery. The site it originates from is set to utf8

Answer 1

The problem is very likely down to not having

use utf8;

(or its equivalent for whatever coding system you are using) in your program. The weird replacements you have there look like problems with bytewise rather than characterwise regular expression replacement.

#!/usr/local/bin/perl
use warnings;
use strict;
use utf8;
binmode STDOUT, "utf8";
my $string = "été";

$string =~ s/[áàâã]/a/gi; #This line always prepends an "a"
$string =~ s/[éèêë]/e/gi;
$string =~ s/[úùûü]/u/gi;

print "$string\n";

prints

ete

If you are reading input from a file or from standard input, make sure you have the stream set to utf8 or whatever is appropriate for the encoding. For STDIN use

binmode STDOUT, "utf8";

If you are reading from a file, use

open my $file, "<:utf8", "file_name"

to get the encoding right. If it is not in UTF-8, use encoding(name) instead of utf8.

Answer 2

Something tells me it's because it doesn't know how to behave with characters with accent. By looking at your Regex, everything seems fine.

Youmight want to check for this : use utf8;

Answer 3

This is probably due to the fact that you're using UTF8 strings, and it's parsing them as if they're not, or similar.

Instead of using something like [áàâã] you should probbaly use something like [\xE1-\xE5]

and probably use use utf8; in your code too

Answer 4

This could also be a problem with Unicode Normalisation, as certain systems (I'm looking at you, OS X) represent extended Latin1 glyphs as a specific normalised representation that can break regular expressions when you refer to a character specifically instead of using a unicode or hex representation.

Answer 5

But did you really want to use regexes at all? Perhaps something like Text::Unidecode would be better

$ perl -Mutf8 -MText::Unidecode -E 'say unidecode("été")'
ete

Answer 6

I suspect that what is happening is that the [áàâã] part of your regex is not actually matching characters, but matching bytes. The UTF-8 encoding of those characters would look literally like this in the regex:

[\xC3\xA1\xC3\xA0\xC3\xA2\xC3\xA3]

And so when the regex is fed, for example , 'é' (\xC3\xA9), it looks at it a byte at a time, matches the \xC3, and replaces it with an 'a'. It does this for all of the \xC3 bytes it can find. So, 'été' is turned into 'a\xA9ta\xA9'.

Then the second regex, which looks like this:

[\xc3\xA9\xC3\xA8\xC3\xAA\xC3\xAB]

comes along, and it matches the \xA9 portion, and replaces it with an 'e'. So now, 'a\xA9ta\xA9' is turned into 'aetae'.

When you replace the [áàâã] with (á|à|â|ã), then that matches complete characters correctly on the first pass, but then your second regex has the original problem, and \xC3 characters are replaced with 'e' instead.

If this is still happening, even with use utf8;, then there may be a bug (or at least a limitation) in the perl regular expression engine. What version of perl are you using for this?

Answer 7

I'd say you shouldn't really use regular expressions here. The easiest way to achieve this (although this might be undesirable) would be to convert your input string into US ASCII. The appropriate conversion tables should know that e is the closest equivalent to é.

Another option would be to use Unicode and normalize your string into NFD. This will break up all accented letters into base letter + diacritic. Then you can just go through your string and remove all combining diacritical characters.