Best way to process large XML in PHP

Question

I have to parse large XML files in php, one of them is 6.5 MB and they could be even bigger. The SimpleXML extension as I've read, loads the entire file into an object, which may not be very efficient. In your experience, what would be the best way?

Answer 1

Check out Pull Parsing in PHP

Answer 2

For a large file, you'll want to use a SAX parser rather than a DOM parser.

With a DOM parser it will read in the whole file and load it into an object tree in memory. With a SAX parser, it will read the file sequentially and call your user-defined callback functions to handle the data (start tags, end tags, CDATA, etc.)

With a SAX parser you'll need to maintain state yourself (e.g. what tag you are currently in) which makes it a bit more complicated, but for a large file it will be much more efficient memory wise.

Answer 3

It really depends on what you want to do with the data? Do you need it all in memory to effectively work with it?

6.5 MB is not that big, in terms of today's computers. You could, for example, ini_set('memory_limit', '128M');

However, if your data can be streamed, you may want to look at using a SAX parser. It really depends on your usage needs.

Answer 4

A SAX Parser, as Eric Petroelje recommends, would be better for large XML files. A DOM parser loads in the entire XML file and allows you to run xpath queries-- a SAX (Simple API for XML) parser will simply read one line at a time and give you hook points for processing.

• SAX Example: http://www.codemiles.com/php-tutorials/php-sax-parser-in-action-t1436.html

Answer 5

SAX parser is the way to go. I've found that SAX parsing can get messy if you don't stay organised.

I use an approach based on STX (Streaming Transformations for XML) to parse large XML files. I use the SAX methods to build a SimpleXML object to keep track of the data in the current context (ie just the nodes between the root and the current node). Other functions are then used for processing the SimpleXML document.

Answer 6

Its very clear that we can use SAX when we can't afford loading the whole tree in the memory. However, what is a good way in SAX; to go to the parent of a particular element; when we hit the startElement handler of an element. for Eg: in what is a good way of accessing and its attribute when sax parser hits ?

Thanks, Aditya

Answer 7

I have heard people having good success with XMLReader: http://us.php.net/manual/en/book.xmlreader.php

Answer 8

I needed to parse a large XML file that happened to have an element on each line (the StackOverflow data dump). In this specific case it was sufficient to read the file one line at a time and parse each line using SimpleXML. For me this had the advantage of not having to learn anything new.