Python UTF-8 can't decode byte on 32-bit machine

Question

it works fine on 64 bit machines but for some reason will not work on python 2.4.3 on a 32-bit instance.

i get the error

'utf8' codec can't decode bytes in position 76-79: invalid data

for the code

try:        
    str(sourceresult.sourcename).encode('utf8','replace')
except:
    raise Exception(  repr(sourceresult.sourcename ) )

it returns 'kazamidori blog\xf9'

i have modified my site.py file to make UTF8 the default encoding, but still doesnt seem to be working.

Answer 1

"Invalid Data" usually means that the incoming data contained characters outside its character set.

This is often caused by, at some point, some data being encoded in a character set different than UTF-8.

For example, if the file a string is stored in was not converted into UTF-8 when you made UTF-8 the standard character set. (In Windows, you can usually specify a file's encoding in the "Save as..." dialog of your text editor)

Or, when data comes from a database that uses a different character set in either the tables, the connection, or both.

Check out where the data comes from, and what encodings are set along the way.

Answer 2

We need the following, and we need the exact output:

type(sourceresult.sourcename) # I suspect it's already a UTF-8 encoded string

repr(sourceresult.sourcename)

Like I said, I'm almost certain that your sourceresult.sourcename is already a UTF-8 encoded string.

Perhaps this might help a little.

EDIT: it seems your sourceresult.sourcename is encoded as cp1252. I don't know what mystring (that you reference in a comment) is. So, to get a UTF-8 encoded string, you need to do:

source_as_UTF8= sourceresult.sourcename.decode("cp1252").encode("utf-8")

However, the string being cp1252-encoded is not consistent with the error message you supplied.

Answer 3

I think the problem is with your use of the str() function. Keep in mind that str() returns narrow, i.e. 1-byte-per-character strings. If the input, sourceresult.sourcename, is unicode, then Python automatically encodes it in order to return a narrow string. By default it uses the system encoding, which is likely something like ISO-8859-1, to do this.

So you're getting the error because it doesn't make sense to call encode on a string that is already encoded. If you get rid of the str(), it should work.