Double-Escaped Unicode Javascript Issue

Question

I am having a problem displaying a Javascript string with embedded Unicode character escape sequences (\uXXXX) where the initial "\" character is itself escaped as "\" What do I need to do to transform the string so that it properly evaluates the escape sequences and produces output with the correct Unicode character?

For example, I am dealing with input such as:

"this is a \u201ctest\u201d";

attempting to decode the "\" using a regex expression, e.g.:

var out  = text.replace('/\/g','\');

results in the output text:

"this is a \u201ctest\u201d";

that is, the Unicode escape sequences are displayed as actual escape sequences, not the double quote characters I would like.

Answer 1

I'm not sure if this is it, but the answer might have something to do with eval(), if you can trust your input.

Answer 2

As it turns out, it's unescape() we want, but with '%uXXXX' rather than '\uXXXX':

unescape(yourteststringhere.replace(/\/g,'%'))

Answer 3

This is a terrible solution, but you can do this:

var x = "this is a \u201ctest\u201d".replace(/\/g,'\\')
// x is now "this is a \u201ctest\u201d"
eval('x = "' + x + '"')
// x is now "this is a “test”"

It's terrible because:

• eval can be dangerous, if you don't know what's in the string

• the string quoting in the eval statement will break if you have actual quotation marks in your string

Answer 4

How many accounts do you have, Jeffrey? I see at least three answers from you in different accounts... (if JW is still you.)

Anyway, I think you can go with the smart solution given by Kev, I tested it quickly in FF3, Opera 9.x, Safari 3 and even IE6! It worked in all browsers: most modern browsers are Unicode aware.
http://www.javascripter.net/faq/unescape.htm -> "(In Unicode-aware browsers, in addition to escape-sequences %XX, the unescape function also processes sequences of the form %uXXXX)"

Test code:

javascript:alert(unescape("this is a \u201ctest\u201d".replace(/\/g, '%')));

Answer 5

(I'm not JW; I have only a single account :) )

Kev's solutions is doing the trick for me. I will evaluate it in other browsers, etc. to see if I encounter any other problems.

Thanks for the solution

Answer 6

I think you are a liar, the account you posted this answer from is not the same one you posted the question.

Jeez, what's will the hostility? There's obviously something going on here that I'm not aware of. Is there some hidden benefit to having multiple accounts that gets people ticked-off for some reason?

I should have said "I don't have any 'accounts'". This is the first question I've ever posted on this site, and simply filled in my name and email address, I never created a specific account.

But I must say, I'm pretty impressed. I was hacking away all morning looking for an answer here and had a solution before I even got up from my desk. Thanks again.

Answer 7

Are you sure '\' is the only character that might get HTML-escaped? Are you sure '\uXXXX' is the only kind of string escape in use?

If not, you'll need a general-purpose HTML-character/entity-reference-decoder and JS-string-literal-decoder. Unfortunately JavaScript has no built-in methods for this and it's quite tedious to do manually with a load of regexps.

It is possible to take advantage of the browser's HTML-decoder by assigning the string to an element's innerHTML property, and then ask JavaScript to decode the string as above:

var el= document.createElement('div');
el.innerHTML= s;
return eval('"'+el.firstChild.data+'"');

However this is an incredibly ugly hack and a security hole if the string comes from a source that isn't 100% trusted.

Where are the strings coming from? It would be nicer if possible to deal with the problem at the server end where you may have more powerful text handling features available. And if you could fix whatever it is that is unnecessarily HTML-escaping your backslashes you could find the problem fixes itself.