Unicode string mess in perl

Question

I have an external module, that is returning me some strings. I am not sure how are the strings returned, exactly. I don't really know, how Unicode strings work and why.

The module should return, for example, the Czech word "být", meaning "to be". (If you cannot see the second letter - it should look like this.) If I display the string, returned by the module, with Data Dumper, I see it as b\x{fd}t.

However, if I try to print it with print $s, I got "Wide character in print" warning, and ? instead of ý.

If I try Encode::decode(whatever, $s);, the resulting string cannot be printed anyway (always with the "Wide character" warning, sometimes with mangled characters, sometimes right), no matter what I put in whatever.

If I try Encode::encode("utf-8", $s);, the resulting string CAN be printed without the problems or error message.

If I use use encoding 'utf8';, printing works without any need of encoding/decoding. However, if I use IO::CaptureOutput or Capture::Tiny module, it starts shouting "Wide character" again.

I have a few questions, mostly about what exactly happens. (I tried to read perldocs, but I was not very wise from them)

1. Why can't I print the string right after getting it from the module?
2. Why can't I print the string, decoded by "decode"? What exactly "decode" did?
3. What exactly "encode" did, and why there was no problem in printing it after encoding?
4. What exactly use encoding do? Why is the default encoding different from utf-8?
5. What do I have to do, if I want to print the scalars without any problems, even when I want to use one of the capturing modules?

edit: Some people tell me to use -C or binmode or PERL_UNICODE. That is a great advice. However, somehow, both the capturing modules magically destroy the UTF8-ness of STDOUT. That seems to be more a bug of the modules, but I am not really sure.

edit2: OK, the best solution was to dump the modules and write the "capturing" myself (with much less flexibility).

Answer 1

1. Because you output a string in perl's internal form (utf8) to a non-unicode filehandle.
2. The decode function decodes a sequence of bytes assumed to be in ENCODING into Perl's internal form (utf8). Your input seems to be already decoded,
3. The encode() function encodes a string from Perl's internal form into ENCODING.
4. The encoding pragma allows you to write your script in any encoding you like. String literals are automatically converted to perl's internal form.
5. Make sure perl knows which encoding your data comes in and come out.

See also perluniintro, perlunicode, Encode module, binmode() function.

Answer 2

You should also look at the PERL_UNICODE environment variable, which is the same as using the -C option. That allows you to set STDIN/STDOUT/STDERR (and @ARGV) to be UTF-8 without having to alter your scripts.

Answer 3

I recommend reading the Unicode chapter of my book Effective Perl Programming. We put together all the docs we could find and explained Unicode in Perl much more coherently than I've seen anywhere else.

This program works fine for me:

#!perl

use utf8;
use 5.010;

binmode STDOUT, ':utf8';

my $string = return_string();

say $string;

sub return_string { 'být' }

Additionally, Capture::Tiny works just fine for me:

#!perl
use utf8;
use 5.010;
use Capture::Tiny qw(capture);

binmode STDOUT, ':utf8';

my( $stdout, $stderr ) = capture {
    system( $^X, '/Users/brian/Desktop/czech.pl' );
    };

say "STDOUT is [$stdout]";

IO::CaptureOutput seems to have some problems though:

#!perl
use utf8;
use 5.010;
use IO::CaptureOutput qw(capture);

binmode STDOUT, ':utf8';

capture {
    system( $^X, '/Users/brian/Desktop/czech.pl' );
    } \my $stdout, \my $stderr;

say "STDOUT is [$stdout]";

For this I get:

STDOUT is [být
]

However, that's easy to fix. Don't use that module. :)