PHP: Problems finding the most frequent character in a UTF-8 string (eg 唐犬土用家犬尨犬山桑)?

Question

From an MySQL database I can extract the following utf-8 characters:

"唐犬土用家犬尨犬山桑山犬巴戦師子幻日幻月引綱忠犬愛犬戌年成犬教条教義"

I am trying to find the most frequent character in this string. I tried putting each as element into an array $arr and do array_count_values($arr); Unfortunately the array operations (or is print_r the culprit?) produce mis-encoded output like this: [0] => � [1] => � [2] => � [3] => � I can display the characters fine in other situations (i.e. retrieving from MySQL and displaying the characters within php works OKAY!), but the array functions (or array output) seem to mess things up.

I HAVE changed /etc/php5/apache2/php.ini
and put default_charset = "utf-8" in there.

(And I HAVE SET NAMES ... etc)

A) Where is the problem? B) Could I do the job without resorting to arrays altogether (i.e. just using string function)?

Thanks for your help.

Answer 1

How are you turning the string into an array? PHP is not multibyte safe by default, so it's probably splitting multibyte characters in the middle.

Remember that in UTF-8 characters are of variable length. Some characters are one byte, some are two, three, or four. (I think up to six is possible, actually.) You would need a very clever string-split algorithm which knew when two or three bytes were one character, and would leave them together.

In the absence of such an algorithm, the simplest solution might be to convert your string into UTF-32. Now all characters are four bytes long, you can split on every four bytes (to the simplistic PHP string functions, that means every four characters, because PHP thinks a byte is a character).

Combining diacritics might still be a problem, though (two Unicode characters making up one printable character). But at least you wouldn't get broken Unicode. You might get COMBINING DIAERESIS on its own, but that's not very broken. I'm not sure how much combining Unicode characters apply to East Asian languages. I'm not a Unicode expert.