Unicode in Perl not working

Question

I have some text files which I am trying to transform with a Perl script on Windows. The text files look normal in Notepad+, but all the regexes in my script were failing to match. Then I noticed that when I open the text files in NotePad+, the status bar says "UCS-2 Little Endia" (sic). I am assuming this corresponds to the encoding UCS-2LE. So I created "readFile" and "writeFile" subs in Perl, like so:

use PerlIO::encoding;

my $enc = ':encoding(UCS-2LE)';

sub readFile {
    my ($fName) = @_;
    open my $f, "<$enc", $fName or die "can't read $fName\n";
    local $/;
    my $txt = <$f>;
    close $f;
    return $txt;
}

sub writeFile {
    my ($fName, $txt) = @_;
    open my $f, ">$enc", $fName or die "can't write $fName\n";
    print $f $txt;
    close $f;
}

my $fName = 'someFile.txt';

my $txt = readFile $fName;
# ... transform $txt using s/// ...
writeFile $fName, $txt;

Now the regexes match (although less often than I expect), but the output contains long strings of Asian-looking characters interspersed with longs strings of the correct text. Is my code wrong? Or perhaps Notepad+ is wrong about the encoding? How should I proceed?

Answer 1

I don't have the Notepad+ editor to check but it may be a BOM problem with your output encoding not containing a BOM.

http://perldoc.perl.org/Encode/Unicode.html#Size%2c-Endianness%2c-and-BOM

Maybe you need to encode $txt using a byte order mark as described above.

Answer 2

OK, I figured it out. The problem was being caused by a disconnect between the encoding translation done by the "encoding..." parameter of the "open" call and the default CRLF translation done by Perl on Windows. What appeared to be happening was that LF was being translated to CRLF on output after the encoding had already been done, which threw off the "parity" of the 16-bit encoding for the following line. Once the next line was reached, the "parity" got put back. That would explain the "long strings of Asian-looking characters interspersed with longs strings of the correct text"... every other line was being messed up.

To correct it, I took out the encoding parameter in my "open" call and added a "binmode" call, as follows:

open my $f, $fName or die "can't read $fName\n";
binmode $f, ':raw:encoding(UCS-2LE)';

binmode apparently has a concept of "layered" I/O handling that is somewhat complicated.

One thing I can't figure out is how to get my CRLF translation back. If I leave out :raw or add :crlf, the "parity" problem returns. I've tried re-ordering as well and can't get it to work.

(I added this as a separate question: http://stackoverflow.com/questions/3320189/crlf-translation-with-unicode-in-perl)