How should escaped unicode be handled by json parsers and encoders?

Question

The json spec allows for escaped unicode in json strings (of the form \uXXXX). It specifically mentions a restricted codepoint (a noncharacter) as a valid escaped codepoint. Doesn't this imply parsers should generate illegal unicode from strings containing noncharacters and restricted codepoints?

An example:

{ "key": "\uFDD0" }

decoding this either requires your parser makes no attempt to interpret the escaped codepoint or that it generates an invalid unicode string. does it not?

Answer 1

What do you mean by “restricted codepoint”? What spec are you looking at that uses that language? (I can't find any such.)

If you are talking about the surrogates then yes: JavaScript knows almost nothing(*) about surrogates and treats all UTF-16 codepoints in any sequence as valid. JSON, being limited to what JavaScript supports, does the same.

*: the only part of JS I can think of that does anything special with surrogates is the encodeURIComponent function, as it uses UTF-8 encoding, in which an attempt to encode an invalid surrogate sequence cannot work. If you try to:

encodeURIComponent('\ud834\udd1e'.substring(0, 1))

you will get an exception.

(Gah! SO seems not to allow characters from outside the Basic Multilingual Plane to be posted directly. Tsk.)

Answer 2

When you decode, it seems that this would be an appropriate use for the unicode replacement character, U+FFFD.

From the Unicode Character Database:

• used to replace an incoming character whose value is unknown or unrepresentable in Unicode
• compare the use of U+001A as a control character to indicate the substitute function