create and stream large XML document in C++

Question

I have some code that creates a fairly large xml DOM and writes it off to a file (up to 50-100MB) . It basically creates the DOM and then calls a toString on it and writes it out with ofstream. Is there a way to get streaming output of the generated dom so that it doesn't create the whole structure in memory all at once and then copy it, etc? I will not modify any node after i create it so it can write it out and free up the memory right away. I could write my own xml class that does the xml construction but ... i don't think that's a good idea since i'll probably miss something when it comes down to escaping etc.

Answer 1

Anytime you're working with large files like this it's probably best to use a memory-mapped file. Create your DOM elements in mapped memory and then commit.

Answer 2

I would recommend GenX as a streaming XML writer, I use this in Programmer's Notepad and it works a treat, you can see examples of use in the source code. Extremely fast, and it produces good UTF-8 XML. Memory usage while you use it should remain roughly constant.

Answer 3

Ok, turns out libxml2 has a streaming API:

http://xmlsoft.org/examples/testWriter.c

It's a little old style (very C-ish) but you can write your wrapper around it.

Answer 4

Look under keyword "C++ XML writer;" XML writers generate XML to file without building the entire DOM in memory so they don't need to use very much memory at all. You didn't mention platform, but Microsoft XmlLite has IXmlWriter.