How to properly escape output (for XHTML) in mako?

Question

Despite offering a nice way to escape output using filters, none of them do the right thing. Taking the string:

x=u"&\u0092"

The filters do the following:

x             Turns the & into an entity but not the \u0092 (valid XML but not XHTML)
h             Exactly the same
u             Escapes both, but obviously uses url escaping
entities      Only converts named entities, so again only the & is escaped
decode.latin1 The same

HTML uses the standard UNICODE Consortium character repertoire, and it leaves undefined (among others) 65 character codes (0 to 31 inclusive and 127 to 159 inclusive)

These seem to be the characters missed. Any ideas?

EDIT

It seems to validate if I use the file offline. Could this be a Content-Type problem?

Answer 1

It is not necessary to convert Unicode characters to the &#xxxx; form to work in HTML unless you're deliberately using the ASCII charset. It's simpler and more efficient to escape named entities, then encode the whole string to UTF-8 and write it out like that. You should probably declare the encoding being used in the HTTP headers or in a <meta> tag.

EDIT:

It seems to validate if I use the file offline. Could this be a Content-Type problem?

Yes. You can either use HTTP headers to enforce a UTF-8 charset or specify it in the HTML directly via a meta tag:

<meta http-equiv="Content-Type" content="application/xhtml+xml;charset=utf-8" />

Answer 2

Validation issues aside, it's useful to be able to remove these characters (which don't display reliably anyway) without necessarily escaping anything else. To this end I added the following function to `lib/helpers.py':

__sgml_invalid = re.compile(r'[\x82-\x8c\x91-\x9c\x9f]', re.UNICODE)

def sgmlsafe(text):
    lookup = {
        130:"‚",    #Single Low-9 Quotation Mark
        131: "ƒ",    #Latin Small Letter F With Hook
        132:"„",    #Double Low-9 Quotation Mark
        133:"…",    #Horizontal Ellipsis
        134:"†",    #Dagger
        135:"‡",    #Double Dagger
        136: "ˆ",    #Modifier Letter Circumflex Accent
        137:"‰",    #Per Mille Sign
        138: "Š",    #Latin Capital Letter S With Caron
        139:"‹",    #Single Left-Pointing Angle Quotation Mark
        140: "Œ",    #Latin Capital Ligature OE
        145:"‘",    #Left Single Quotation Mark
        146:"’",    #Right Single Quotation Mark
        147:"“",    #Left Double Quotation Mark
        148:"”",    #Right Double Quotation Mark
        149:"•",    #Bullet
        150:"–",    #En Dash
        151:"—",    #Em Dash
        152: "˜",    #Small Tilde
        153:"™",    #Trade Mark Sign
        154: "š",    #Latin Small Letter S With Caron
        155:"›",    #Single Right-Pointing Angle Quotation Mark
        156: "œ",    #Latin Small Ligature OE
        159: "Ÿ"     #Latin Capital Letter Y With Diaeresis
        }

    return __sgml_invalid.sub(lambda x: lookup[ord(x.group())], text)

And you can make this available as a filter by editing environment.py:

config['pylons.app_globals'].mako_lookup = TemplateLookup(
    ...
    imports=[....,'from appname.lib.helpers import sgmlsafe',...]

It should then be available to your templates:

${c.content|n,sgmlsafe}