views:

1048

answers:

4

I'm writing some mail-processing software in Python that is encountering strange bytes in header fields. I suspect this is just malformed mail; the message itself claims to be us-ascii, so I don't think there is a true encoding, but I'd like to get out a unicode string approximating the original one without throwing a UnicodeDecodeError .

So, I'm looking for a function that takes a str and optionally some hints and does its darndest to give me back a unicode. I could write one of course, but if such a function exists its author has probably thought a bit deeper about the best way to go about this.

I also know that Python's design prefers explicit to implicit and that the standard library is designed to avoid implicit magic in decoding text. I just want to explicitly say "go ahead and guess".

+8  A: 

You may be interested in Universal Encoding Detector.

insin
+1 for being 5 seconds faster than me :-)
unbeknown
Really useful, thanks. But not in the standard library.
Nick
+4  A: 

+1 for the chardet module (suggested by insin).

It is not in the standard library, but you can easily install it with the following command:

$ easy_install chardet

Example:

>>> import chardet
>>> import urllib
>>> detect = lambda url: chardet.detect(urllib.urlopen(url).read())
>>> detect('http://stackoverflow.com')
{'confidence': 0.85663169917190185, 'encoding': 'ISO-8859-2'}    
>>> detect('http://stackoverflow.com/questions/269060/is-there-a-python-lib')
{'confidence': 0.98999999999999999, 'encoding': 'utf-8'}

See Installing EasyInstall if you don't have one.

J.F. Sebastian
@J.F. Sebastian: Didn't it strike you that `ISO-8859-2` was a nonsense?
John Machin
@John Machin: Yes, it was. It is educational to show that you should not blindly trust it. Current results are different ('utf-8' and 'ascii' correspondingly).
J.F. Sebastian
A: 

The best way to do this that I've found is to iteratively try decoding a prospective with each of the most common encodings inside of a try except block.

Jeremy Cantrell
+1  A: 

As far as I can tell, the standard library doesn't have a function, though it's not too difficult to write one as suggested above. I think the real thing I was looking for was a way to decode a string and guarantee that it wouldn't throw an exception. The errors parameter to string.decode does that.

def decode(s, encodings=('ascii', 'utf8', 'latin1')):
    for encoding in encodings:
     try:
      return s.decode(encoding)
     except UnicodeDecodeError:
      pass
    return s.decode('ascii', 'ignore')
Nick