Memory management while loading huge XML files

Question

We have an application which imports objects from an XML. The XML is around 15 GB. The application invariably starts running out of memory. We tried to free memory in between operations but this has lead to degrading performance. i.e it takes more time to complete the import operation. The CPU utilization reaches 100%

The application is written in C++.

Does the frequent call to free() will lead to performance issues?

---

Promoted from a comment by the OP: the parser being used in expat, which is a SAX parser with a very small footprint, and customisable memory management.

Answer 1

Use SAX parser instead of DOM parser.

Answer 2

Have you tried resuing the memory and your classes as opposed to freeing and reallocating it? Constant allocation/deallocation cycles, especially if they are coupled with small (less than 4096 bytes) data fragments can lead to serious performance problems and memory address space fragmentation.

Answer 3

Profile the application during one of these bothersome loads, to see where it is spending most of its time.

I believe that free() can sometimes be costly, but that is of course very much dependent on the platform's implementation.

Also, you don't say a lot about the lifelength of the objects loaded; if the XML is 15 GB, how much of that is kept around for each "object", once the markup is parsed and thrown away?

It sounds sensible to process an input document of this size in a streaming fashion, i.e. not trying a DOM-approach which loads and builds the entire XML parse tree at once.

Answer 4

We tried to free memory in between operations but this has lead to degrading performance .

Does the frequent call to free() will lead to performance issues ?

Based on the evidence supplied, yes.

Since your already using expat, a SAX parser, what exactly are you freeing? If you can free it, why are you mallocing it in a loop in the first place?

Answer 5

Maybe, it should say profiler.

Also don't forget that work with heap is single-thread. I mean that if booth of your threads will allocate/free memory in ont time, one of them will waiting when first will done.

If you allocating and free memory for same objects, you could create pool of this object and do allocate/free once.

Answer 6

Try and find a way to profile your code.

If you have no profiler, try and organize your work so that you only have a few free() commands (instead of the many you suggest).

It is very common to have to find the right balance between memory consumption and time efficiency.

Answer 7

If you want to minimise your memory usage, took a look at How to read the XML data from a file by using Visual C++.

Answer 8

I did not try it myself, but have you heard of XMLLite, there's an MSDN artical introducing it. It's used by MS Office internally.

Answer 9

One thing that often helps is to use a lightweight low-overhead memory pool. If you combine this with "frame" allocation methods (ignoring any delete/free until you're all done with the data), you can get something that's ridiculously fast.

We did this for an embedded system recently, mostly for performance reasons, but it saved a lot of memory as well.

The trick was basically to allocate a big block -- slightly bigger than we'd need (you could allocate a chain of blocks if you like) -- and just keep returning a "current" pointer (bumping it up by allocSize, rounded up to maximum align requirement of 4 in our case, each time). This cut our overhead per alloc from on the order of 52-60 bytes down to <= 3 bytes. We also ignored "free" calls until we were all done parsing and then freed the whole block.

If you're clever enough with your frame allocation you can save a lot of space and time. It might not get you all the way to your 15GiB, but it would be worth looking at how much space overhead you really have... My experience with DOM-based systems is that they use tons of small allocs, each with a relatively high overhead.

(If you have virtual memory, a large "block" might not even hurt that much, if your access at any given time is local to a page or three anyway...)

Obviously you have to keep the memory you actually need in the long run, but the parser's "scratch memory" becomes a lot more efficient this way.