UTF8 Filenames in PHP and Different Unicode Encodings

Question

I have a file containing Unicode characters on a server running linux. If I SSH into the server and use tab-completion to navigate to the file/folder containing unicode characters I have no problem accessing the file/folder. The problem arises when I try accessing the file via PHP (the function I was accessing the file system from was stat). If I output the path generated by the PHP script to the browser and paste it into the terminal the file also seems to exist (even though looking at the terminal the file paths are exactly the same).

I set PHP to use UTF8 as its default encoding via php_ini as well as set mb_internal_encoding. I checked the PHP filepath string encoding and it comes out as UTF8, as it should. Poking around a bit more I decided to hexdump the é character that the terminal's tab-completion and compare it to the hexdump of the 'regular' é character created by the PHP script or by manually entering in the character via keyboard (option+e+e on os x). Here is the result:

echo -n é | hexdump
0000000 cc65 0081                              
0000003
echo -n é | hexdump
0000000 a9c3                                   
0000002

The é character that allows a correct file reference in the terminal is the 3-byte one. I'm not sure where to go from here, what encoding should I use in PHP? Should I be converting the path to another encoding via iconv or mb_convert_encoding?

Answer 1

The three byte sequence is actually the utf8 representation of an e (0x65) followed by a combining ´ (0xcc 0x81), while 0xc3 0xa9 stands "directly" for é.
An utf-8 aware collation should be aware of the possible decompositions, but I don't know how you can enable that (and probably recompile the php source) on a mac.
Best I can offer is the "Using UTF-8 with Gentoo" description.

Answer 2

Firstly: You should try to avoid imposing semantics on the names of files. I can't really tell why PHP is generating filenames in your scenario, so I can't suggest how you should apply this rule.

The different (two byte and three byte) representations of é are UTF-8 encodings of the composed and decomposed variations of this character in Unicode. In Unicode these are distinct ways to represent the same visual character. Unicode has the concept of "canonicalisation" in which all representations of the same character are converted to a single representation, sort of like squashing two strings to lowercase to perform a caseless comparison.

Linux does not perform canonicalisation or any other processing automatically for file names, so a file may be named with precomposed (like the two byte sequence) or decomposed (like the three byte sequence) characters or any mix of the two, it's up to whoever named the file. If you are creating the files, you could set a policy (e.g. always use precomposed characters) and write some code to enforce it. Otherwise, you can't rely on any particular rule here.

Answer 3

Thanks to the tips given in the two answers I was able to poke around and find some methods for normalizing the different unicode decompositions of a given character. In the situation I was faced with I was accessing files created by a OS X Carbon application. It is a fairly popular application and thus its file names seemed to adhere to a specific unicode decomposition.

In PHP 5.3 a new set of functions was introduced that allows you to normalize a unicode string to a particular decomposition. Apparently there are four decomposition standards which you can decompose you unicode string into. Python has had unicode normalization capabilties since version 2.3 via unicode.normalize. This article on python's handling of unicode strings was helpful in understanding encoding / string handling a bit better.

Here is a quick example on normalizing a unicode filepath:

filePath = unicodedata.normalize('NFD', filePath)

I found that the NFD format worked for all my purposes, I wonder if this is this is the standard decomposition for unicode filenames.