Python unicode problems on Windows XP

Question

Having the following django view code that generates a CSV response from a database view:

def _get_csv_stats(request, **filterargs):
   result = GlobalStats.objects.select_related().filter(**filterargs).values_list('user__username',
                                                                                  'user__first_name','user__last_name',
                                                                                  'center__name', 'action_name',
                                                                                  'action_date').annotate(num = Count('id')).order_by("action_date")
   response = HttpResponse(mimetype = 'text/csv')
   response.write( codecs.BOM_UTF8 )
   response['Content-Disposition'] = 'attachment; filename=statistcs.csv'
   writer = UnicodeWriter(response)
   for value in result:
       writer.writerow([unicode(v) for v in value])
   return response

Some of the columns contain utf8 text. When I download the file and open it using Linux or Mac OS X I can see the text properly. But downloading and opening the file using windows XP strange characters appear in the place of the non-ASCII text. Converting the file from csv to xls using Open Office on linux and opening the file on windows xp will result with a readable file (no strange characters).

I can't see what I'm missing here as I don't work with win XP. Has anyone experienced anything similar?

The UnicodeWriter class I'm using is descibed here at the bottom.

Answer 1

Whether the text is displayed properly depends on whether the font that is is being drawn in supports all Unicode characters. This is not a problem with your django.

Answer 2

First: I don't know django, so maybe this is far off.

Check whether your writer actually outputs UTF-8, if you write an UTF-8 BOM into the file. Unicode and UTF-8 are not the same. Also, afaikt, does Windows Notepad understand a UTF-8 BOM, so you getting "strange" characters can mean two things:

a) the rest of the file is not really UTF-8-encoded, try v.encode('utf-8') for that

b) the characters your trying to display are not supported by the font