How do I perform HTML decoding/encoding using Python/Django?

Question

I have a string that is html encoded:

<img class="size-medium wp-image-113" 
  style="margin-left: 15px;" title="su1" 
  src="http://blah.org/wp-content/uploads/2008/10/su1-300x194.jpg" 
  alt="" width="300" height="194" />

I want to change that to:

<img class="size-medium wp-image-113" style="margin-left: 15px;" 
  title="su1" src="http://blah.org/wp-content/uploads/2008/10/su1-300x194.jpg" 
  alt="" width="300" height="194" />

I want this to register as HTML so that it is rendered as an image by the browser instead of being displayed as text.

I've found how to do this in C# but not in in Python. Can someone help me out?

Thanks.

Edit: Someone asked why my strings are stored like that. It's because I am using a web-scraping tool that "scans" a web-page and gets certain content from it. The tool (BeautifulSoup) returns the string in that format.

Related

• Convert XML/HTML Entities into Unicode String in Python

Answer 1

The question should be, why are you strings stored like that?

EDIT: I don't know why this is being down voted. It's a valid question, you're adding to the file size and adding zero security if you're just going to re-encode them.

Answer 2

I found this in the Cheetah source code (here)

htmlCodes = [
    ['&', '&'],
    ['<', '&lt;'],
    ['>', '&gt;'],
    ['"', '&quot;'],
]
htmlCodesReversed = htmlCodes[:]
htmlCodesReversed.reverse()
def htmlDecode(s, codes=htmlCodesReversed):
    """ Returns the ASCII decoded version of the given HTML string. This does
        NOT remove normal HTML tags like <p>. It is the inverse of htmlEncode()."""
    for code in codes:
        s = s.replace(code[1], code[0])
    return s

not sure why they reverse the list, I think it has to do with the way they encode, so with you it may not need to be reversed. Also if I were you I would change htmlCodes to be a list of tuples rather than a list of lists... this is going in my library though :)

i noticed your title asked for encode too, so here is Cheetah's encode function.

def htmlEncode(s, codes=htmlCodes):
    """ Returns the HTML encoded version of the given string. This is useful to
        display a plain ASCII text string on a web page."""
    for code in codes:
        s = s.replace(code[0], code[1])
    return s

Answer 3

The Cheetah function should work, but is missing the single-quote. Use this tuple instead:

htmlCodes = (
    ('&', '&'),
    ('<', '&lt;'),
    ('>', '&gt;'),
    ('"', '&quot;'),
    ("'", '&#39;'),
)

Here's Django's django.utils.html.escape function for reference:

def escape(html):
    """Returns the given HTML with ampersands, quotes and carets encoded."""
    return mark_safe(force_unicode(html).replace('&', '&amp;').replace('<', '&l
t;').replace('>', '&gt;').replace('"', '&quot;').replace("'", '&#39;'))

I also think it would make more sense to store the HTML unescaped in your database. It'd be worth looking into getting unescaped results back from BeautifulSoup if possible.

In addition, escaping only occurs in Django during template rendering. So to prevent escaping you just tell the templating engine not to escape your string: Use either {{ context_var|safe }} or {% autoescape off %}{{ context_var }}{% endautoescape %} in your templates.

Answer 4

Try regular expressions (like preg_replace in php).

Answer 5

Use daniel's solution if the set of encoded characters is relatively restricted. Otherwise, use one of the numerous HTML-parsing libraries.

I like BeautifulSoup because it can handle malformed XML/HTML :

http://www.crummy.com/software/BeautifulSoup/

for your question, there's an example in their documentation

from BeautifulSoup import BeautifulStoneSoup
BeautifulStoneSoup("Sacré bleu!", 
                   convertEntities=BeautifulStoneSoup.HTML_ENTITIES).contents[0]
# u'Sacr\xe9 bleu!'

Answer 6

See at the bottom of this page at Python wiki, there are at least 2 options to "unescape" html.

Answer 7

For html encoding, there's cgi.escape from the standard library:

>> help(cgi.escape)
cgi.escape = escape(s, quote=None)
    Replace special characters "&", "<" and ">" to HTML-safe sequences.
    If the optional flag quote is true, the quotation mark character (")
    is also translated.

For html decoding, I use the following:

from htmlentitydefs import name2codepoint
# for some reason, python 2.5.2 doesn't have this one (apostrophe)
name2codepoint['#39'] = 39

def unescape(s):
    "unescape HTML code refs; c.f. http://wiki.python.org/moin/EscapingHtml"
    return re.sub('&(%s);' % '|'.join(name2codepoint),
              lambda m: unichr(name2codepoint[m.group(1)]), s)

For anything more complicated, I use BeautifulSoup.

Answer 8

Daniel's comment as an answer:

"escaping only occurs in Django during template rendering. Therefore, there's no need for an unescape - you just tell the templating engine not to escape. either {{ context_var|safe }} or {% autoescape off %}{{ context_var }}{% endautoescape %}"

Answer 9

I found a fine function at: http://snippets.dzone.com/posts/show/4569

def decodeHtmlentities(string):
    import re
    entity_re = re.compile("&(#?)(\d{1,5}|\w{1,8});")

    def substitute_entity(match):
        from htmlentitydefs import name2codepoint as n2cp
        ent = match.group(2)
        if match.group(1) == "#":
            return unichr(int(ent))
        else:
            cp = n2cp.get(ent)

            if cp:
                return unichr(cp)
            else:
                return match.group()

    return entity_re.subn(substitute_entity, string)[0]

Answer 10

slowkvant's solution was the only solution I could find that would unescape quotations but I don't have a high enough reputation to vote it up and apparently to comment on it. So I am posting this. Thanks slowkvant.