Django: Getting a Python encoding error when handling HTTP response in Latin1?

Question

I'm working in Django, and using urllib2 and simplejson to parse some information from an API.

The problem is that the API returns information in the Latin-1 encoding, and just once in a while there's a character in there that causes Django to crash horribly with an encoding error. This is my code:

get_person_id_url = "http://www.domain.com/api/get?" + \
    "key=KEY&num="+ urllib2.quote(number) + "&always_return=true&output=js"
request = urllib2.Request(get_person_id_url, None, {'Referer': ''})
response = urllib2.urlopen(request)
results = json.load(response)
person_id = results["person_id"]

I know that I can use something like this to turn Latin1 strings into UTF8:

responseString = responseString.decode('latin1').encode('utf-8')

but it seems this only works on strings, so I'm not completely sure how or where to apply it in the above code. What should I decode and where to trap any errors before they occur?

Unfortunately I don't remember what API call to make to return a character that will crash Django - so I can't do much testing before it goes live. I'm hoping StackOverflow can help... Thanks!

Answer 1

Call response.read() to get the data of the response. Then in a try/except do your latin1 decoding and json loading. In general, Django should never crash if you wrap potentially error-causing operations in exception handlers and take care of them appropriately (at least log them somewhere so you can deal with them at some point in the future).