Python Unicode ajax form posting error

Question

Hey, i got a n00b problem with python and i've been searching here for a while and i couldnt find a proper solution...

i got a utf8 form that i ajax post to a python page. i read the json simplejson with utf-8 charset.

the text is fine as long as there is no mixed utf8 and latin chars like ?!;, etc...

UnicodeDecodeError: 'ascii' codec can't decode byte 0xd7 in position 0: ordinal not in range(128)

commentjsonarray = simplejson.loads(commentjson, encoding='utf-8')

i tried a bunch of things but i cant get it to work. help.

just an update for you with more code for help, thanks

commentjson = request.POST['commentObj']

commentjsonarray = simplejson.loads(commentjson, encoding='utf-8')
program = get_object(Program, commentjsonarray['programid'])
userget = get_object(User, commentjsonarray['userid'])
#get user avatar from usermeta
usermeta = get_object(UserMeta, 'user_id = ',userget.key())
commenttext = commentjsonarray['walltext']
from django.utils.encoding import smart_unicode,force_unicode,smart_str

commenttext = smart_str(commenttext)

newcomment = db_create(Wall, user_avatarurl=str(usermeta.avatarurlsmall),user_fullname=str(''+userget.first_name+' '+userget.last_name),user_idstring=str(userget.key()),text = str(commenttext) , program_id = program.key() , user_id = userget.key())

above is the python part. here is the javascript:

var walltext = $('walltext').value

 var commentObj = {"walltext": ""+walltext+"", "programid": programid, "userid": userid};
 var commentJSON = encodeURIComponent(Object.toJSON(commentObj));
if (walltext != '' || walltext == 'type here' || walltext.length > 0) {

    new Ajax.Request('/wall/new', {
        method: 'post',
        encoding: 'UTF-8',
        parameters: 'commentObj=' + commentJSON,
        onSuccess: function(request){
            var msg = request.responseText.evalJSON();
            if (msg) {

                var structure = '<div id="' + msg.msgid + '"><img src="' + msg.avatarurl + '" width="18" height="18"> ' + msg.username + ':' + msg.text + '<div id="frontSepLine"></div></div>';

                //$('programwall').insert({bottom:structure});
                refreshWall(msg.programid);

                $('walltext').value = 'type here';

                var objDiv = document.getElementById("programwall");
                objDiv.scrollTop = objDiv.scrollHeight;


            }



        }
    });

Answer 1

The situation you've described works fine for me (with the standard library json on Python 2.6), either with or without the explicit encoding (which is not needed for a utf-8 encoded bytestring, as utf-8 is the default here):

>> s = u'{"valá":"macché?!"}'.encode('utf8')
>>> json.loads(s)
{u'val\xe1': u'macch\xe9?!'}
>>> json.loads(s, encoding='utf-8')
{u'val\xe1': u'macch\xe9?!'}

and also with simplejson 2.1.1 (really redundant on 2.6, but, oh well;-):

>>> import simplejson
>>> s = u'{"valá":"macché?!"}'.encode('utf8')
>>> simplejson.loads(s)
{u'val\xe1': u'macch\xe9?!'}
>>> simplejson.loads(s, encoding='utf-8')
{u'val\xe1': u'macch\xe9?!'}

Can you describe your problem more accurately? Your description of what triggers your error, i.e. the reverse of "as long as there is no mixed utf8 and latin chars like ?!;, etc", doesn't cause any problem -- so what about showing us the tiniest example that does reproduce your problem?