Values in UTF-8 being encoded as NULL in JSON

Question

I have a set of keywords that are passed through via JSON from a DB (encoded UTF-8), some of which may have special characters like é, è, ç, etc. This is used as part of an auto-completer. Example:

array('Coffee', 'Cappuccino', 'Café');

I should add that the array as it comes from the DB would be:

array('Coffee', 'Cappuccino', 'Café');

But JSON encodes as:

["coffee", "cappuccino", null];

If I print these via print_r(), they show up fine on a UTF-8 encoded webpage, but café comes through as "café" if text/plain is used if I want to look at the array using print_r($array);exit();.

If I encode using utf8_encode() before encoding to JSON, it comes through fine, but what gets printed on the webpage is "café" and not "café".

Also strange, but json_last_error() is being seen as an undefined function, but json_decode() and json_encode() work fine.

Any ideas on how to get UTF-8 encoded data from the database to behave the same throughout the entire process?

EIDT: Here is the PHP function that grabs the keywords and makes them into a single array:

private function get_keywords() 
{
    global $db, $json;

    $output = array();

    $db->query("SELECT keywords FROM listings");

    while ($r = $db->get_array())
    {
        $split = explode(",", $r['keywords']);

        foreach ($split as $s)
        {
            $s = trim($s);
            if ($s != "" && !in_array($s, $output)) $output[] = strtolower($s);
        }
    }

    $json->echo_json($output);
}

The json::echo_json method just encodes, sets the header and prints it (for usage with Prototype)

EDIT: DB Connection method:

function connect()
{

    if ($this->set['sql_connect'])
    {
        $this->connection = @mysql_connect( $this->set['sql_host'], $this->set['sql_user'], $this->set['sql_pass'])
                OR $this->debug( "Connection Error", mysql_errno() .": ". mysql_error());
        $this->db = @mysql_select_db( $this->set['sql_name'], $this->connection)
                OR $this->debug( "Database Error", "Cannot Select Database '". $this->set['sql_name'] ."'");

        $this->is_connected = TRUE;
    }

    return TRUE;
}

More Updates: Simple PHP script I ran:

echo json_encode( array("Café") ); // ["Caf\u00e9"]
echo json_encode( array("Café") ); // null

Answer 1

Try sending your array through this function before doing json_encode():

<?php

function utf8json($inArray) {

    static $depth = 0;

    /* our return object */
    $newArray = array();

    /* safety recursion limit */
    $depth ++;
    if($depth >= '30') {
        return false;
    }

    /* step through inArray */
    foreach($inArray as $key=>$val) {
        if(is_array($val)) {
            /* recurse on array elements */
            $newArray[$key] = utf8json($inArray);
        } else {
            /* encode string values */
            $newArray[$key] = utf8_encode($val);
        }
    }

    /* return utf8 encoded array */
    return $newArray;
}
?>

Taken from comment on phpnet @ http://php.net/manual/en/function.json-encode.php.

The function basically loops though array elements, perhaps you did your utf-8 encode on the array itself?

Answer 2

json_encode seems to be dropping strings that contain invalid characters. It is likely that your UTF-8 data is not arriving in the proper form from your database.

Looking at the examples you give, my wild guess would be that your database connection is not UTF-8 encoded and serves ISO-8859-1 characters instead.

Can you try a SET NAMES utf8; after initializing the connection?

Answer 3

I tried your code sample like this

[~]> cat utf.php 
<?php
$arr = array('Coffee', 'Cappuccino', 'Café');
print json_encode($arr);
[~]> php utf.php 
["Coffee","Cappuccino","Caf\u00e9"]
[~]>

Based on that I would say that if the source data is really UTF-8, then json_encode works just fine. If its not, then thats where you get null. Why its not, I cannot tell based on this information.