Python Unicode CSV export (using Django)

Question

Hi All,

I'm using a Django app to export a string to a CSV file. The string is a message that was submitted through a front end form. However, I've been getting this error when a unicode single quote is provided in the input.

UnicodeEncodeError: 'ascii' codec can't encode character u'\u2019' 
  in position 200: ordinal not in range(128)

I've been trying to convert the unicode to ascii using the code below, but still get a similar error.

UnicodeEncodeError: 'ascii' codec can't encode characters in 
position 0-9: ordinal not in range(128)

I've sifted through dozens of websites and learned a lot about unicode, however, I'm still not able to convert this unicode to ascii. I don't care if the algorithm removes the unicode characters. The commented lines indicate some various options I've tried, but the error persists.

import csv
import unicodedata

...

#message = unicode( unicodedata.normalize(
#                            'NFKD',contact.message).encode('ascii','ignore'))
#dmessage = (contact.message).encode('utf-8','ignore')
#dmessage = contact.message.decode("utf-8")
#dmessage = "%s" % dmessage
dmessage = contact.message

csv_writer.writerow([
        dmessage,
])

Does anyone have any advice in removing unicode characters to I can export them to CSV? This seemingly easy problem has kept my head spinning. Any help is much appreciated. Thanks, Joe

Answer 1

Encoding is a pain, but if you're working in django have you tried smart_unicode(str) from django.utils.encoding? I find that usually does the trick.

The only other option I've found is to use the built-in python encode() and decode() for strings, but you have to specify the encoding for those and honestly, it's a pain.

Answer 2

You can't encode the Unicode character u'\u2019' (U+2019 Right Single Quotation Mark) into ASCII, because ASCII doesn't have that character in it. ASCII is only the basic Latin alphabet, digits and punctuation; you don't get any accented letters or ‘smart quotes’ like this character.

So you will have to choose another encoding. Now normally the sensible thing to do would be to export to UTF-8, which can hold any Unicode character. Unfortunately for you if your target users are using Office (and they probably are), they're not going to be able to read UTF-8-encoded characters in CSV. Instead Excel will read the files using the system default code page for that machine (also misleadingly known as the ‘ANSI’ code page), and end up with mojibake like ’ instead of ’.

So that means you have to guess the user's system default code page if you want the characters to show up correctly. For Western users, that will be code page 1252. Users with non-Western Windows installs will see the wrong characters, but there's nothing you can do about that (other than organise a letter-writing campaign to Microsoft to just drop the stupid nonsense with ANSI already and use UTF-8 like everyone else).

Code page 1252 can contain U+2019 (’), but obviously there are many more characters it can't represent. To avoid getting UnicodeEncodeError for those characters you can use the ignore argument (or replace to replace them with question marks).

dmessage= contact.message.encode('cp1252', 'ignore')

alternatively, to give up and remove all non-ASCII characters, so that everyone gets an equally bad experience regardless of locale:

dmessage= contact.message.encode('ascii', 'ignore')

Answer 3

[caveat: I'm not a djangoist; django may have a better solution].

General non-django-specific answer:

If you have a smallish number of known non-ASCII characters and there are user-acceptable ASCII equivalents for them, you can set up a translation table and use the unicode.translate method:

smashcii = {
    0x2019 : u"'",
    # etc
    #

smashed = input_string.translate(smashcii)