How to handle user input of invalid UTF-8 characters?

Question

I'm looking for general a strategy/advice on how to handle invalid UTF-8 input from users.

Even though my webapp uses UTF-8, somehow some users enter invalid characters. This causes errors in PHP's json_encode() and overall seems like a bad idea to have around.

W3C I18N FAQ: Multilingual Forms says "If non-UTF-8 data is received, an error message should be sent back.".

• How exactly should this be practically done, throughout a site with dozens of different places where data can be input?
• How do you present the error in a helpful way to the user?
• How do you temporarily store and display bad form data so the user doesn't lose all their text? Strip bad characters? Use a replacement character, and how?
• For existing data in the database, when invalid UTF-8 data is detected, should I try to convert it and save it back (how? utf8_encode()? mb_convert_encoding()?), or leave as-is in the database but doing something (what?) before json_encode()?

EDIT: I'm very familiar with the mbstring extension and am not asking "how does UTF-8 work in PHP". I'd like advice from people with experience in real-world situations how they've handled this.

EDIT2: As part of the solution, I'd really like to see a fast method to convert invalid characters to U+FFFD

Answer 1

There is a multibyte extension for PHP, check it out: http://www.php.net/manual/en/book.mbstring.php

You should try mb_check_encoding() function.

Good luck!

Answer 2

Receiving invalid characters from your web app might have to do with the character sets assumed for HTML forms. You can specify which character set to use for forms with the accept-charset attribute:

<form action="..." accept-charset="UTF-8">

You also might want to take a look at similar questions in StackOverflow for pointers on how to handle invalid characters, e.g. those in the column to the right, but I think that signaling an error to the user is better than trying to clean up those invalid characters which cause unexpected loss of significant data or unexpected change of your user's inputs.

Answer 3

How about stripping all chars outside your given subset. At least in some parts of my application I would not allow using chars outside the [a-Z] [0-9 sets], for example usernames. You can build a filter function that strips silently all chars outside this range, or that returns an error if it detects them and pushes the decision to the user.

Answer 4

Try doing what Rails does to force all browsers always to post UTF-8 data:

<form accept-charset="UTF-8" action="#{action}" method="post"><div
    style="margin:0;padding:0;display:inline">
    <input name="utf8" type="hidden" value="&#x2713;" />
  </div>
  <!-- form fields -->
</form>

See railssnowman.info or the initial patch for an explanation.

1. To have the browser sends form-submission data in the UTF-8 encoding, just render the page with a Content-Type header of "text/html; charset=utf-8" (or use a meta http-equiv tag).
2. To have the browser sends form-submission data in the UTF-8 encoding, even if the user fiddles with the page encoding (browsers let users do that), use accept-charset="UTF-8" in the form.
3. To have the browser sends form-submission data in the UTF-8 encoding, even if the user fiddles with the page encoding (browsers let users do that), and even if the browser is IE and the user switched the page encoding to Korean and entered Korean characters in the form fields, add a hidden input to the form with a value such as ✓ which can only be from the Unicode charset (and, in this example, not the Korean charset).

Answer 5

The accept-charset="UTF-8" attribute is only a guideline for browsers to follow, they are not forced to submit that in that way, crappy form submission bots are a good example...

What I usually do is ignore bad chars, either via iconv() or with the less reliable utf8_encode() / utf8_decode() functions, if you use iconv you also have the option to transliterate bad chars.

Here is an example using iconv():

$str_ignore = iconv('UTF-8', 'UTF-8//IGNORE', $str);
$str_translit = iconv('UTF-8', 'UTF-8//TRANSLIT', $str);

If you want to display an error message to your users I'd probably do this in a global way instead of a per value received basis, something like this would probably do just fine:

function utf8_clean($str)
{
    return iconv('UTF-8', 'UTF-8//IGNORE', $str);
}

$clean_GET = array_map('utf8_clean', $_GET);

if (serialize($_GET) != serialize($clean_GET))
{
    $_GET = $clean_GET;
    $error_msg = 'Your data is not valid UTF-8 and has been stripped.';
}

// $_GET is clean!

You may also want to normalize new lines and strip (non-)visible control chars, like this:

function Clean($string, $control = true)
{
    $string = iconv('UTF-8', 'UTF-8//IGNORE', $string);

    if ($control === true)
    {
        return preg_replace('~\p{C}+~u', '', preg_replace('~\r[\n]?~', "\n", $string));
    }

    return preg_replace('~[^\P{C}\t\n]+~u', '', preg_replace('~\r[\n]?~', "\n", $string));
}

---

Code to convert from UTF-8 to Unicode codepoints:

function Codepoint($char)
{
    $result = null;
    $codepoint = unpack('N', iconv('UTF-8', 'UCS-4BE', $char));

    if (is_array($codepoint) && array_key_exists(1, $codepoint))
    {
        $result = sprintf('U+%04X', $codepoint[1]);
    }

    return $result;
}

echo Codepoint('à'); // U+00E0
echo Codepoint('ひ'); // U+3072

Probably faster than any other alternative, haven't tested it extensively though.

---

Example:

$string = 'hello world�';

// U+FFFEhello worldU+FFFD
echo preg_replace_callback('/[\p{So}\p{Cf}\p{Co}\p{Cs}\p{Cn}]/u', 'Bad_Codepoint', $string);

function Bad_Codepoint($string)
{
    $result = array();

    foreach ((array) $string as $char)
    {
        $codepoint = unpack('N', iconv('UTF-8', 'UCS-4BE', $char));

        if (is_array($codepoint) && array_key_exists(1, $codepoint))
        {
            $result[] = sprintf('U+%04X', $codepoint[1]);
        }
    }

    return implode('', $result);
}

Is this what you were looking for?

Answer 6

I recommend merely not allowing garbage to get in. Don't rely on custom functions, which can bog your system down. Simply walk the submitted data against an alphabet you design. Create an acceptable alphabet string and walk the submitted data, byte by byte, as if it were an array. Push acceptable characters to a new string, and omit unacceptable characters. The data you store in your database then is data triggered by the user, but not actually user-supplied data.

EDIT #4: Replacing bad character with entiy: �

EDIT #3: Updated : Sept 22 2010 @ 1:32pm Reason: Now string returned is UTF-8, plus I used the test file you provided as proof.

<?php
// build alphabet
// optionally you can remove characters from this array

$alpha[]= chr(0); // null
$alpha[]= chr(9); // tab
$alpha[]= chr(10); // new line
$alpha[]= chr(11); // tab
$alpha[]= chr(13); // carriage return

for ($i = 32; $i <= 126; $i++) {
$alpha[]= chr($i);
}

/* remove comment to check ascii ordinals */

// /*
// foreach ($alpha as $key=>$val){
//  print ord($val);
//  print '<br/>';
// }
// print '<hr/>';
//*/
// 
// //test case #1
// 
// $str = 'afsjdfhasjhdgljhasdlfy42we875y342q8957y2wkjrgSAHKDJgfcv kzXnxbnSXbcv   '.chr(160).chr(127).chr(126);
// 
// $string = teststr($alpha,$str);
// print $string;
// print '<hr/>';
// 
// //test case #2
// 
// $str = ''.'©?™???';
// $string = teststr($alpha,$str);
// print $string;
// print '<hr/>';
// 
// $str = '©';
// $string = teststr($alpha,$str);
// print $string;
// print '<hr/>';

$file = 'http://www.cl.cam.ac.uk/~mgk25/ucs/examples/UTF-8-test.txt';
$testfile = implode(chr(10),file($file));

$string = teststr($alpha,$testfile);
print $string;
print '<hr/>';


function teststr(&$alpha, &$str){
    $strlen = strlen($str);
    $newstr = chr(0); //null
    $x = 0;
    if($strlen >= 2){

        for ($i = 0; $i < $strlen; $i++) {
            $x++;
            if(in_array($str[$i],$alpha)){
                // passed
                $newstr .= $str[$i];
            }else{
                // failed
                print 'Found out of scope character. (ASCII: '.ord($str[$i]).')';
                print '<br/>';
                $newstr .= '&#65533;';
            }
        }
    }elseif($strlen <= 0){
        // failed to qualify for test
        print 'Non-existent.';

    }elseif($strlen === 1){
        $x++;
        if(in_array($str,$alpha)){
            // passed

            $newstr = $str;
        }else{
            // failed
            print 'Total character failed to qualify.';
            $newstr = '&#65533;';
        }
    }else{
        print 'Non-existent (scope).';
        }

if(mb_detect_encoding($newstr, "UTF-8") == "UTF-8"){
// skip
}else{
    $newstr = utf8_encode($newstr);
}


// test encoding:
if(mb_detect_encoding($newstr, "UTF-8")=="UTF-8"){
    print 'UTF-8 :D<br/>';
    }else{
        print 'ENCODED: '.mb_detect_encoding($newstr, "UTF-8").'<br/>';
        }




return $newstr.' (scope: '.$x.', '.$strlen.')';
}

Answer 7

I put together a fairly simple class to check if input is in UTF-8 and to run through utf8_encode() as needs be:

class utf8
{

    /**
     * @param array $data
     * @param int $options
     * @return array
     */
    public static function encode(array $data)
    {
        foreach ($data as $key=>$val) {
            if (is_array($val)) {
                $data[$key] = self::encode($val, $options);
            } else {
                if (false === self::check($val)) {
                    $data[$key] = utf8_encode($val);
                }
            }
        }

        return $data;
    }

    /**
     * Regular expression to test a string is UTF8 encoded
     * 
     * RFC3629
     * 
     * @param string $string The string to be tested
     * @return bool
     * 
     * @link http://www.w3.org/International/questions/qa-forms-utf-8.en.php
     */
    public static function check($string)
    {
        return preg_match('%^(?:
            [\x09\x0A\x0D\x20-\x7E]              # ASCII
            | [\xC2-\xDF][\x80-\xBF]             # non-overlong 2-byte
            |  \xE0[\xA0-\xBF][\x80-\xBF]        # excluding overlongs
            | [\xE1-\xEC\xEE\xEF][\x80-\xBF]{2}  # straight 3-byte
            |  \xED[\x80-\x9F][\x80-\xBF]        # excluding surrogates
            |  \xF0[\x90-\xBF][\x80-\xBF]{2}     # planes 1-3
            | [\xF1-\xF3][\x80-\xBF]{3}          # planes 4-15
            |  \xF4[\x80-\x8F][\x80-\xBF]{2}     # plane 16
            )*$%xs',
            $string);
    }
}

// For example
$data = utf8::encode($_POST);

Answer 8

For completeness to this question (not necessarily the best answer)...

function as_utf8($s) {
    return mb_convert_encoding($s, "UTF-8", mb_detect_encoding($s));
}