ansaurus

Question

How to decode string to use with Google Language Detection API?

Answer 1

+1 A:

Maybe urllib.unquote is what you are looking for:

>>> from urllib import unquote
>>> unquote("%DD%F2%EE%20%F2%E5%F1%F2")

This gives you a string in which the characters are in whatever encoding that you've used in the URL. If you want to recode it to a different encoding (say, UTF-8), you have to create a unicode object first and then use the encode method of the unicode object to recode it:

>>> from urllib import unquote, quote
>>> import json, urllib2, pprint
>>> decoded = unicode(unquote("%DD%F2%EE%20%F2%E5%F1%F2"), "windows-1251")
>>> print decoded
Это тест
>>> recoded = decoded.encode("utf-8")

At this point, we have an UTF-8 encoded string, but this is still not suitable to be passed on to the Google Language Detection API:

>>> recoded
'\xd0\xad\xd1\x82\xd0\xbe \xd1\x82\xd0\xb5\xd1\x81\xd1\x82'

Since you want to include this string in a URL as a query argument, you have to encode it using urllib.quote:

>>> url = "http://ajax.googleapis.com/ajax/services/language/detect?v=1.0&amp;q=%s" % quote(recoded)
>>> data = json.loads(urllib2.urlopen(url).read())
>>> pprint.pprint(data)
{u'responseData': {u'confidence': 0.094033934,
                   u'isReliable': False,
                   u'language': u'ru'},
 u'responseDetails': None,
 u'responseStatus': 200}

Tamás 2010-10-12 12:51:19

Looks good when I try to print it, but when I send it to Google it throws exception: UnicodeEncodeError: 'ascii' codec can't encode characters in position

Orsol 2010-10-12 13:15:21

You have to pass `recoded` on to `urllib.quote` to obtain a representation which can safely be appended to the Google Language API URL. I'm modifying my example to show that.

Tamás 2010-10-12 13:31:53

ansaurus

tags:

views:

answers:

How to decode string to use with Google Language Detection API?

related questions