Python templates for huge HTML/XML

Question

Hello,

Recently I needed to generate a huge HTML page containing a report with several thousand row table. And, obviously, I did not want to build the whole HTML (or the underlying tree) in memory. As result, I built the page with the old good string interpolation, but I do not like the solution.

Thus, I wonder whether there are Python templating engines that can yield resulting page content by parts.

UPD 1: I am not interested in listing all available frameworks and templating engines. I am interested in templating solutions that I can use separately from any framework and which can yield content by portions instead of building the whole result in memory.

I understand the usability enhancements from partial content loading with client scripting, but that is out of the scope of my current question. Say, I want to generate a huge HTML/XML and stream it into a local file.

Answer 1

It'd be more user-friendly (assuming they have javascript enabled) to build the table via javascript by using e.g. a jQuery plugin which allows automatical loading of contents as soon as you scroll down. Then only few rows are loaded initially and when the user scrolls down more rows are loaded on demand.

If that's not a solution, you could use three templates: one for everything before the rows, one for everything after the rows and a third one for the rows. Then you first send the before-rows template, then generate the rows and send them immediately, then the after-rows template. Then you will have only one block/row in memory instead of the whole table.

Answer 2

Are you using a web framework for this? http://www.pylonshq.com includes compatibility with several templating engines. http://www.djangoproject.com/ Django has its own templating language.

I think an answer that included lazy loading of the rows with javascript would work for web view, but I presume the report is going to need to be printed, in which case you'll have to build the whole thing at some point, right?

Answer 3

There is no problem with building something like this in memory. Several thounsand rows is by no means big.

For your templating needs you can use any of the:

• rst http://docutils.sourceforge.net/rst.html (just needs core docutils)
• markdown http://daringfireball.net/projects/markdown/ (python lib)
• sphinx http://sphinx.pocoo.org/ (python based)
• json http://www.json.org/ (python lib)

There are some tools that allow generation of HTML from these markup languages.

Answer 4

Most popular template engines have a way to generate or write rendered result to file objects with chunks. For example:

• Template.generate() in Jinja2
• Template.render_context() in Mako
• Stream.serialize() in Genshi

Answer 5

You don't need a streaming templating engine - I do this all the time, and long before you run into anything vaguely heavy server-side, the browser will start to choke. Rendering a 10000 row table will peg the CPU for several seconds in pretty much any browser; scrolling it will be bothersomely choppy in chrome, and the browser mem usage will rise regardless of browser.

What you can do (and I've previously implemented, even though in retrospect it turns out not to be necessary) is use client-side xslt. Printing the xslt processing instruction and the opening and closing tag using strings is easy and fairly safe; then you can stream each individual row as a standalone xml element using whatever xml writer technique you prefer.

However - you really don't need this, and likely never will - if ever your html generator gets too slow, the browser will be an order of magnitude more problematic.

So, unless you benchmarked this and have determined you really have a problem, don't waste your time. If you do have a problem, you can solve it without fundamentally changing the method - in memory generation can work just fine.