Hashing JSON in PHP does not produce the same result as in Javascript for unicode characters

Question

Hey!

My web application communicates with the server over JSON protocol. Before sending each JSON message from the web application, I run a hmac-sha1 function on it (on already encoded object) and insert the resulting HMAC into the header of JSON request.

On server side, I decode JSON message with PHP, extract the HMAC, unset() the HMAC from the object and then encode the object back into JSON and create a HMAC of it.

The HMACs match as long as I don't use characters like "ž, š, č". When I use those characters in the message, the HMACs don't match anymore.

In the web application I'm using jQuery.post() to transmit the already encoded JSON string.

If I send the data I got from the web application back to it in the JSON encoded reply, the application will display "ž, č, š" just nicely.

How can I make the HMACs match?

UPDATE: This is only a problem on latest version of Firefox and Opera. It works fine on IE8 and Chrome. On the former browsers, the JSON string (before it is sent) is:

{"body":[{"name":"Žiga Kraljevič","email":"[email protected]","password":"secretpass"}],"header":{"apiID":"person-27jhfa83ha-js84sjj18dasjd","hmac":"e4259d6ef8f477c020d644409cc16dd9c42301e8"}}

While on the latter browsers (IE8 and Chrome, where it works) is the following:

{"body":[{"name":"\u017diga Kraljevi\u010d","email":"[email protected]","password":"secretpass"}],"header":{"apiID":"person-27jhfa83ha-js84sjj18dasjd","hmac":"e4e9e2d0d8d11728a2b4329ad6dacdb9409b1de1"}}

Answer 1

You're probably running into multiple issues. One of them may well be that the character encoding being used on the client is different from that being used on the server, worth ensuring that they're the same (more about character encoding in Joel's excellent essay). Another may well be that there are multiple correct ways to encode things. The encoders may well be using different ways. For instance, you can encode a " within a string as either \" or \u0022. Both are valid, and they're equivalent, but the hashes won't match. Similarly, I'm a bit surprised you're not running into more trouble when not using accented characters, for instance with whitespace.

Answer 2

What is your hmac-sha1 function, where's it from? If it is taking a JSON String as input then there's an implicit encode-to-bytes step going on here because SHA1 operates on bytes, not UTF-16 code units like JS String.

I would suspect that your JS function is using a “one code unit n per byte n” type of encoding, for easy calculation with tools like getCharCodeAt. This is effectively the same as if the character string input had been encoded to ISO-8859-1. Whereas if you are using encodeURIComponent or posting the raw characters via XMLHttpRequest, the implicit encoding there is UTF-8.

You could convert the String to UTF-8-bytes-stored-as-code-units format for the JS hmac-sha1 function, that might make it match PHP. There's a sneaky idiom to do this:

var utf8= unescape(encodeURIComponent(s));

When POSTing JSON I base64 and urlencode it anyway

URL-encoding should be enough (with encodeURIComponent, not escape which is the wrong thing for absolutely everything except the reverse step of the UTF-8-conversion trick above).

BTW, what's the purpose of this? You do know it doesn't in any way secure the connection between the browser and the server, yeah?

Edit:

I'm using jssha.sourceforge.net for sha1-hmac. In PHP I'm using hash_hmac.

Works for me:

var data= '\u017E, \u010D, \u0161'; // 'ž, č, š' in a Unucode string
var utf8bytes= unescape(encodeURIComponent(data));
var hmac= new jsSHA(utf8bytes).getHMAC('foo', 'ASCII', 'SHA-1', 'HEX');
alert(hmac); // 5d15f0b9...
var form= 'message='+encodeURIComponent(data)+'&hmac='+encodeURIComponent(hmac);
xmlhttprequest.send(form);

...

$utf8bytes= $_POST['message']; // "\xc5\xbe, \xc4\x8d, \xc5\xa1"
                               // which is 'ž, č, š' as UTF-8 in byte string
$hmac= hash_hmac('sha1', $utf8bytes, 'foo');
echo $hmac; // 5d15f0b9...
echo strtolower($hmac)===strtolower($_POST['hmac']); // true

This uses the binary ('ASCII' to jsSHA) key foo. If you are using a binary key with non-ASCII characters in it, you would have to make sure that those are properly encoded too, in the same way as the data.

The key for HMAC is a shared secret between the server and the client, which has been previously exchanged over a secure connection.

It's not only the key you'd have to send over a secure connection, but the entire page and all scripts in it. Otherwise a man in the middle attack could sabotage your scripts on the way to the browser to replace them with a version that used the secret key to sign bogus messages. If you've got an HTTPS server for all this stuff, fine. I'm not sure what the HMAC would be doing in that case though, it seems a bit involved for an anti-XSRF scheme.