views:

597

answers:

4

I'm currently persisting filenames in a sqlite database for my own purposes. Whenever I try to insert a file that has a special character (like é etc.), it throws the following error:

pysqlite2.dbapi2.ProgrammingError: You must not use 8-bit bytestrings unless you use a text_factory that can interpret 8-bit bytestrings (like text_factory = str). It is highly recommended that you instead just switch your application to Unicode strings.

When I do "switch my application over to Unicode strings" by wrapping the value sent to pysqlite with the unicode method like: unicode(filename), it throws this error:

UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 66: ordinal not in range(128)

Is there something I can do to get rid of this? Modifying all of my files to conform isn't an option.

UPDATE If I decode the text via filename.decode("utf-8"), I'm still getting the ProgrammingError above.

My actual code looks like this:

cursor.execute("select * from musiclibrary where absolutepath = ?;",
    [filename.decode("utf-8")])

What should my code here look like?

+1  A: 

You need to specify the encoding of filename for conversion to Unicode, for example: filename.decode('utf-8'). Just using unicode(...) picks the console encoding, which is often unreliable (and often ascii).

Nicholas Riley
I tried doing that, but it seems that I'm still getting the errors mentioned above. I updated the post with what I'm doing now, so you can see what I'm doing. Thanks!
TK Kocheran
My bad, I had some more bad conversion happening later on in my script that was throwing the same error :)
TK Kocheran
+1  A: 

Read this. It's important.

Ignacio Vazquez-Abrams
Very helpful in understanding where Unicode sits in Python.
TK Kocheran
A: 

Have you tried to pass the unicode string directly:

cursor.execute("select * from musiclibrary where absolutepath = ?;",(u'namé',))

You will need to add the file encoding at the beginning of the script:

# coding: utf-8
newtover
If I try that, it seems to be working. I'm iterating over around 3000 files, and it fails on a filename like: 02 - Neighborhood #2 (Laïka).mp3 . Is there a conversion technique that I'm missing somewhere?
TK Kocheran
+1  A: 

You should pass as Unicode the arguments of your SQL statement.

Now, it all depends on how you obtain the filename list. Perhaps you're reading the filesystem using os.listdir or os.walk? If that is the case, there is a way to have directly the filenames as Unicode just by passing a Unicode argument to either of these functions:
Examples:

  • os.listdir(u'.')
  • os.walk(u'.')

Of course, you can substitute the u'.' directory with the actual directory whose contents you are reading. Just make sure it's a Unicode string.

ΤΖΩΤΖΙΟΥ
Very helpful, thanks!
TK Kocheran