Is Django double encoding a Unicode (utf-8?) string?

Answer 1

This does seem like a case of double-encoding; I don't have much experience with Python, but try adjusting the MySQL connection settings as per the advice at http://tahpot.blogspot.com/2005/06/mysql-and-python-and-unicode.html

What I'm guessing is happening is that the connection is latin1, so MySQL tries to encode the string again before storage to the UTF-8 field. The code there, specifically this bit:

EDIT: With Python when establishing a database connection add the following flag: init_command='SET NAMES utf8'.

In addition set the following in MySQL's my.cnf: default-character-set = utf8

is probably what you want.

Answer 2

I added set names utf8 to my php data insertion sequence, and now in a Python shell the feared ndash shows up as \x96. This renders correctly when read and output through Django.

One unusual situation about this is that I'm inserting data through PHP. Django issues set names utf8 automatically, so likely if I was inserting and reading the data through Django, this issue would not have appeared. PHP was using the default of latin1, I suppose

As an interesting note, while before I could read the data from PHP and it showed up normally in the browser, now the ndash is � unless I call set namesbefore reading the data.

So, it's working now and I hope I never have to understand whatever was going on before!