tags:

views:

106

answers:

3

I have an external module, that is returning me some strings. I am not sure how are the strings returned, exactly. I don't really know, how Unicode strings work and why.

The module should return, for example, the Czech word "být", meaning "to be". (If you cannot see the second letter - it should look like this.) If I display the string, returned by the module, with Data Dumper, I see it as b\x{fd}t.

However, if I try to print it with print $s, I got "Wide character in print" warning, and ? instead of ý.

If I try Encode::decode(whatever, $s);, the resulting string cannot be printed anyway (always with the "Wide character" warning, sometimes with mangled characters, sometimes right), no matter what I put in whatever.

If I try Encode::encode("utf-8", $s);, the resulting string CAN be printed without the problems or error message.

If I use use encoding 'utf8';, printing works without any need of encoding/decoding. However, if I use IO::CaptureOutput or Capture::Tiny module, it starts shouting "Wide character" again.

I have a few questions, mostly about what exactly happens. (I tried to read perldocs, but I was not very wise from them)

  1. Why can't I print the string right after getting it from the module?
  2. Why can't I print the string, decoded by "decode"? What exactly "decode" did?
  3. What exactly "encode" did, and why there was no problem in printing it after encoding?
  4. What exactly use encoding do? Why is the default encoding different from utf-8?
  5. What do I have to do, if I want to print the scalars without any problems, even when I want to use one of the capturing modules?

edit: Some people tell me to use -C or binmode or PERL_UNICODE. That is a great advice. However, somehow, both the capturing modules magically destroy the UTF8-ness of STDOUT. That seems to be more a bug of the modules, but I am not really sure.

edit2: OK, the best solution was to dump the modules and write the "capturing" myself (with much less flexibility).

+5  A: 
  1. Because you output a string in perl's internal form (utf8) to a non-unicode filehandle.
  2. The decode function decodes a sequence of bytes assumed to be in ENCODING into Perl's internal form (utf8). Your input seems to be already decoded,
  3. The encode() function encodes a string from Perl's internal form into ENCODING.
  4. The encoding pragma allows you to write your script in any encoding you like. String literals are automatically converted to perl's internal form.
  5. Make sure perl knows which encoding your data comes in and come out.

See also perluniintro, perlunicode, Encode module, binmode() function.

eugene y
Thanks a lot. How do I open STDOUT so it's a unicode filehandle? And why it is not in default? edit: Oh right, I see, it's the binmode. The second "sub-question" still stands. Why is binmode of STDOUT not UTF-8 by default?
Karel Bílek
@Karel Bilek: Probably it's not the default because of backwards-compatibility issues. Perl6 gets this right though.
Daenyth
No, sadly, it doesn't work. The capturing functions (both of them) make binmode of STDOUT non-utf8 again, thus breaking it.
Karel Bílek
@Karel Bílek: because when it was the default (in 5.8.0 only, and only when the user's locale was a utf8 one, IIRC) it broke a lot of stuff for a lot of people.
ysth
+1  A: 

You should also look at the PERL_UNICODE environment variable, which is the same as using the -C option. That allows you to set STDIN/STDOUT/STDERR (and @ARGV) to be UTF-8 without having to alter your scripts.

cjm
Nope. Even the `-C` doesn't survive the capturing functions. But it seems to be more the problem with the functions themselves, and not Perl. I guess.
Karel Bílek
OK. The easiest resolution... was to write the function myself (with a lot less flexibility). Thanks :)
Karel Bílek
@Karel Bílek, what value were you using with `-C`? There are a number of possible settings, as explained in the docs I linked to.
cjm
+2  A: 

I recommend reading the Unicode chapter of my book Effective Perl Programming. We put together all the docs we could find and explained Unicode in Perl much more coherently than I've seen anywhere else.

This program works fine for me:

#!perl

use utf8;
use 5.010;

binmode STDOUT, ':utf8';

my $string = return_string();

say $string;

sub return_string { 'být' }

Additionally, Capture::Tiny works just fine for me:

#!perl
use utf8;
use 5.010;
use Capture::Tiny qw(capture);

binmode STDOUT, ':utf8';

my( $stdout, $stderr ) = capture {
    system( $^X, '/Users/brian/Desktop/czech.pl' );
    };

say "STDOUT is [$stdout]";

IO::CaptureOutput seems to have some problems though:

#!perl
use utf8;
use 5.010;
use IO::CaptureOutput qw(capture);

binmode STDOUT, ':utf8';

capture {
    system( $^X, '/Users/brian/Desktop/czech.pl' );
    } \my $stdout, \my $stderr;

say "STDOUT is [$stdout]";

For this I get:

STDOUT is [být
]

However, that's easy to fix. Don't use that module. :)

brian d foy
FWIW, IO::CaptureOutput "works" for me. But I consider that buggy and the output you get to be correct. qx// works how I consider to be correct, giving you 5 characters captured (including the newline) by default and 4 characters when `use open IN=>":utf8";` is specified.
ysth