Detect file encoding in PHP

Question

I have a script which combines a number of files into one, and it breaks when one of the files has UTF8 encoding. I figure that I should be using the utf8_decode() function when reading the files, but I don't know how to tell which need decoding.

My code is basically:

$output = '';
foreach ($files as $filename) {
    $output .= file_get_contents($filename) . "\n";
}
file_put_contents('combined.txt', $output);

Currently, at the start of a UTF8 file, it adds these characters in the output: 

Answer 1

How are you going to handle the non-ascii characters from the UTF-8 or 16 or 32 file?

I ask because I think you may have a design issue here.

I would convert your output file into UTF-8 (or 16 or 32) instead of the other way around.

Then you won't have this problem.

Have you also considered the security issues that may arise from converting an escaped UTF8 code? See this comment:

Detecting multi-byte encoding

Figure out what encoding your source file is in, then convert it to UTF8 and you should be good to go.

Answer 2

Try using the mb_detect_encoding function. This function will examine your string and attempt to "guess" what its encoding is. You can then convert it as desired. As brulak suggested, however, you're probably better off converting to UTF-8 rather than from, to preserve the data you're transmitting.

Answer 3

Your answer is here : http://en.wikipedia.org/wiki/Utf8#Byte-order_mark

Choose utf8 without BOM for PHP.