Encoding detection library in python

views:

226

answers:

+2 Q:

Encoding detection library in python

This is somehow related to my question here.

I process tons of texts (in HTML and XML mainly) fetched via HTTP. I'm looking for a library in python that can do smart encoding detection based on different strategies and convert texts to unicode using best possible character encoding guess.

I found that chardet does auto-detection extremely well. However auto-detecting everything is the problem because it is SLOW and very much against all standards. As per chardet FAQ I don't want to screw the standards.

From the same FAQ here is the list of places where I want to look for encoding:

charset parameter in HTTP Content-type header.
<meta http-equiv="content-type"> element in the <head> of a web page for HTML documents.
encoding attribute in the XML prolog for XML documents.
Auto-detect the character encoding as a last resort.

Basically I want to be able to look in all those place and also deal with conflicting information automatically.

Is there such library out there or do I need to write it myself?

+2 A:

BeautifulSoup (the html parser) incorporates a class called UnicodeDammit that does just that. Have a look and see if you like it.

Ranieri 2010-02-21 22:58:15

It looks very promising. Thank you!

parxier 2010-02-21 23:06:44

+2 A:

BeautifulSoup's UnicodeDammit, which in turn uses chardet.

chardet by itself is quite useful for the general case (determining text's encoding) but slow as you say. UnicodeDammit adds extra features on top of chardet, in particular that it can look up the encoding explicitly specified in XML's encoding tags.

As for the HTTP Content-type header, I think you need to read that yourself to extract the charset parameter, and then pass it to UnicodeDammit in the fromEncoding parameter.

As for resolving conflicts, UnicodeDammit will give precedence to explicitly-stated encoding (if the encoding doesn't generate errors). See the docs for full details.

Craig McQueen 2010-02-21 23:52:44

ansaurus

tags:

views:

answers:

Encoding detection library in python

related questions