Efficient merging of multiple, large xml files into one

Question

I searched the web and I searched stackoverflow up and down. No solution. Although I found solutions how to do this within pure xslt here.

But the problem is that the resulting xml will be several hundred MB large. So I must do this with SAX in Java. (please no xslt solution, although I tagged it with xslt ;-))

Let me explain with more detail. I have several multiple xml files (preferable InputSteam's) which should be parsed. The files or InputStream's looks like

inputstream1

<root>
  <doc>
    <tag>test1</tag>
  </doc>
  <doc>
    <tag>test2</tag>
  </doc>
  ...
</root>

inputstream2

<root>
  <doc>
    <tag>test3</tag>
  </doc>
  <doc>
    <tag>test4</tag>
  </doc>
  ...
</root>

inputstream1+inputstream2+...+inputstreamN = resulting xml. It will look like

<root>
  <doc>
    <tag>test1</tag>
  </doc>
  <doc>
    <tag>test2</tag>
  </doc>
  ...
   <doc>
    <tag>test3</tag>
  </doc>
  <doc>
    <tag>test4</tag>
  </doc>
  ...
</root>

Do someone has a solution or a link for this? Is this possible via implementing a custom InputSource or should I use a custom ContentHandler? Or is this possible with joost/stx?

The nice thing if I could use a ContentHandler would be that I could apply some minor transformations (I already implemented this). But then the problem is that I don't know a way to pass multiple files or InputStream's as InputSource:

XMLReader xmlReader = XMLReaderFactory.createXMLReader();
xmlReader.setContentHandler(customHandler);
xmlReader.parse(getInputSource()); // only one InputStream possible

or should I parse the InputStreams directly within my ContentHandler?

Answer 1

You may want to have a look at the pay-for version of Saxon. It can handle on-the-fly XSLT not needing the full DOM in memory.

Answer 2

I haven't done this myself, but I recalled seeing an IBM developerworks article that looked like it made this pretty easy.

It's a bit old now, but try http://www.ibm.com/developerworks/xml/library/x-tipstx5/index.html

This is StAX instead of SAX. I'm not sure current JDKs include StAX. If not you can probably get it from http://stax.codehaus.org/

Answer 3

I finally managed this via the following snippet:

  finalHandler = new StreamResult(new OutputStreamWriter(System.out));
  // customHandler extends DefaultHandler
  CustomTransformerHandler customHandler = new CustomTransformerHandler(
         finalHandler);
  customHandler.startDocumentExplicitly();
  InputStream is = null;
  while ((is = customHandler.createNextInputStream()) != null) {
    // multiple inputStream parsing
    XMLReader myReader = XMLReaderFactory.createXMLReader();
    myReader.setContentHandler(customHandler);
    myReader.parse(new InputSource(is));
  }
  customHandler.endDocumentExplicitly();

The important part was to leave the startDocument and endDocument methods empty. All other methods (characters, startElement, endElement) will be redirected to the finalHandler. The customHandler.createNextInputStream method returns null if all inputstreams are read.

Answer 4

the most effective way to merge files are to use byte level cut and paste feature offered by VTD-XML, AFAIK. You take both files, parse them into VTDNav objects, then instantiate an XMLModifier object, grab the fragments from the second file, and insert them into the first file... that got to be far more efficient than SAX..