How do you convert HTML entities to Unicode and vice versa in Python?

Question

Duplicate: Convert XML/HTML Entities into Unicode String in Python

How do you convert HTML entities to Unicode and vice versa in Python?

Answer 1

You need to have BeautifulSoup.

from BeautifulSoup import BeautifulStoneSoup
import cgi

def HTMLEntitiesToUnicode(text):
    """Converts HTML entities to unicode.  For example '&' becomes '&'."""
    text = unicode(BeautifulStoneSoup(text, convertEntities=BeautifulStoneSoup.ALL_ENTITIES))
    return text

def unicodeToHTMLEntities(text):
    """Converts unicode to HTML entities.  For example '&' becomes '&'."""
    text = cgi.escape(text).encode('ascii', 'xmlcharrefreplace')
    return text

text = "&, ®, <, >, ¢, £, ¥, €, §, ©"

uni = HTMLEntitiesToUnicode(text)
htmlent = unicodeToHTMLEntities(uni)

print uni
print htmlent
# &, ®, <, >, ¢, £, ¥, €, §, ©
# &amp;, &#174;, &lt;, &gt;, &#162;, &#163;, &#165;, &#8364;, &#167;, &#169;

Answer 2

Can also be done without external libraries. See http://stackoverflow.com/questions/663058/html-entity-codes-to-text/663128#663128

Answer 3

As to the "vice versa" (which I needed myself, leading me to find this question, which didn't help, and subsequently another site which had the answer):

u'some string'.encode('ascii', 'xmlcharrefreplace')

will return a plain string with any non-ascii characters turned into XML (HTML) entities.