ANSI, ASCII, Unicode and encoding confusion with Python

Question

Hi! I was happily using BeautifulSoup and I'm also using a text file as input parameters of my Python script.

I then came across the famous "UnicodeEncodeError" error.

I've been reading questions here at SO but I'm still confused.

What does ASCII got to do with all of these? What encoding do I use on my text editor (Notepad++)? ANSI? UTF-8? Decoding a string to ASCII doesn't seem to always work (I'm guessing the string is in a different encoding coming from BeautifulSoup). How do I fix this?

Anyway any help and clarifications will be greatly appreciated.

Thanks!

edit: reading BeautifulSoup's docs, it says that it only uses unicode but I'm still getting Unicode errors :(

  File "C:\Python26\lib\encodings\cp437.py", line 12, in encode
    return codecs.charmap_encode(input,errors,encoding_map)
UnicodeEncodeError: 'charmap' codec can't encode character u'\u300d' in position
 3: character maps to <undefined>

Answer 1

ANSI is not a character encoding (in common parlance it refers to certain escape sequences, though it's of course the acronym for the American National Standard Institute). You can set the encoding in Notepad++ (and check what encoding you're using) -- hopefully utf-8, because that's a universal encoding (lets you represent any Unicode point). You build unicode from your utf-8 encoded text with an explicit decode method call, or you read the file as unicode with a codecs.open (both require you to specify your encoding name -- again, hopefully 'utf8').

Answer 2

What does ASCII got to do with all of these?

Python has no way to find out what encoding was used to store text, so it assumes ascii by default. However, ASCII defines only first 128 chars, so anything outside results in decode error (which is actually good thing, since it does not let you use incorrectly decoded strings around).

Most of the time your string would be in utf-8, since its most common way to encode Unicode, so its usually safe to do s.decode('utf-8') on str type strings (or use unicode(s, 'utf-8') call)

If you dont know in advance what kind of encoding text has, and it provides no encoding metadata, you can try using chardet module.

BeautifulSoup can output result in different encodings and ways, so you just need to specify that you want unicode there.