tags:

views:

205

answers:

3

I have a textfile encoded in UTF-16. Each line contains a number of columns separated by tabs. For those who care, the file is a playlist TXT export from iTunes. Column #27 contains a filename.

I am reading it using Perl 5.8.8 in Linux using code similar to:

binmode STDIN, ":encoding(UTF-16)";
while(<>)
{
    chomp;
    my @cols = split /\t/, $_;
    my $filename = $cols[26];   # Column #27 contains the filename
    print "File exists!" if (-e "$filename");
}

(Please note: I've shortened this code snippet. In my actual code I do some substitutions to convert the absolute windows filename used by iTunes into a filename valid on my Linux box)

Even though the files exist, the (-e) file test does not return true. I believe it has something to do with the string being in UTF-16 but cannot figure out what the problem is. The actual filename uses only ASCII characters. And the filename prints correctly if I print the $filename variable.

Can filenames in Perl be in UTF16? Any ideas how to get this code snippet to work?

+2  A: 

If, as you say, the actual filename uses only ASCII characters, wouldn't

$filename =~ s/\0//g;

work? Anyway, xxd should help the next time you run into something like this

[sinan@archardy ~]$ xxd /mnt/c/Documents\ and\ Settings/sinan/Desktop/test.txt
0000000: fffe 2f00 6800 6f00 6d00 6500 2f00 7300  ../.h.o.m.e./.s.
0000010: 6900 6e00 6100 6e00 2f00 7400 6500 7300  i.n.a.n./.t.e.s.
0000020: 7400 6d00 6500 2e00 7400 7800 7400 0d00  t.m.e...t.x.t...
0000030: 0a00                                     ..

I see that you have solved your problem in the time it took me to create a test file and reboot into Linux. Oh well.

Sinan Ünür
You would think. But it does not. -e still returns false.Just to test the rest of my code, I tried hardcoding a filename inside the Perl file, and it worked. Reading from the iTunes UTF16 file (even with your null substitution suggestion) does not work.
blt04
Try utf8:downgrade($filename) before the null substitution.
Inshallah
@Inshalla: Still doesn't work.
blt04
Well let's see some debugging then, what's actually inside $filename, byte by byte?
bobince
Thanks again Sinan. I finally saw the 0d0a when looking more closely via xxd.
blt04
+5  A: 

The UTF-16 text is processed by the :encoding layer. By the time it gets into $_, there's no way to tell that it was ever UTF-16. I don't think that's your issue.

My guess would be that you've either got some whitespace in your filename (that you didn't notice when you tried printing it out) or you're not in the directory you think you are.

Try

if (-e $filename) { print "File exists!" } 
else { print "File <$filename> not found" }

and check the filename carefully. You might also use Cwd; and print out the current directory.

cjm
Thanks cjm: I saw this after I posted my solution, but you were correct.
blt04
+3  A: 

I figured out the solution:

Column 27 is the last column, and the file is encoded with 0d0a (\r\n) line endings. chomp was only removing 0a (\n). Not sure why I didn't see this before, but it doesn't have anything to do with UTF16.

Adding:

s/\r$//;

after chomp fixes the problem.

Thanks for your help - sorry to send you down a rabbit trail.

blt04
So I was right, it's whitespace in your filenames :-)
cjm
You could also try `:crlf:encoding(UTF-16)`, although I've never tried using :crlf with UTF-16, so I'm not sure if that works. I've only used :crlf with single-byte encodings.
cjm