How to completely sanitize a string of illegal characters in python?

Question

I have a feature of my program where the user can upload a csv file, which my program goes through and uses as input. I have one user complaining about a problem where his input is throwing up an error. The error is cause by there being an illegal character that is encoded wrong. The characters is below:

�

Sometimes it appears as a diamond with a "?" in the middle. Sometimes it appears as a double diamond with "?" in the middle, sometimes it appears as "\xa0", and sometimes it appears as "\xa0\xa0".

In my program if I do:

print str_with_weird_char

the striong will show up in my terminal with the diamond "?" in place of the weird character. If I copy+paste that string into ipython, it will exit with this message:

In [1]: g="blah��blah"
WARNING: 
********
You or a %run:ed script called sys.stdin.close() or sys.stdout.close()!
Exiting IPython!

notice how the diamond "?" is double now. For some reason copy+paste makes it double...

In the django traceback page, it looks like this:

UnicodeDecodeError at /chris/import.html
('ascii', 'blah \xa0 BLAH', 14, 15, 'ordinal not in range(128)')

The thing that messes me up is that I can't do anything with this string without it throwing an exceotion. I tried unicode(), I tried str(), I tried .encode(), I tried .encode("utf-8"), no matter what it throws up an error.

What can I do it get this thing to be a working string?

Answer 1

You can pass, "ignore" to skip invalid characters in .encode/.decode like "ILLEGAL".decode("utf8","ignore")

>>> "ILLEGA\xa0L".decode("utf8")
...
UnicodeDecodeError: 'utf8' codec can't decode byte 0xa0 in position 6: unexpected code byte

>>> "ILLEGA\xa0L".decode("utf8","ignore")
u'ILLEGAL'
>>>