Big XML file and OutOfMemoryError

Question

Hello, I’m trying to parse a XML file up to 500 mb in java. I tried to use SAX but it gives me this error java.lang.OutOfMemoryError: Java heap space at com.sun.org.apache.xerces.internal.util.XMLStringBuffer.append(Unknown Source) Can you help me? Thanks a lot. P.S. Smaller XML files works just fine

Answer 1

try using Streaming API for XML (new in java6) its made for doing this

http://www.javabeat.net/articles/14-java-60-features-part-2-pluggable-annotation-proce-2.html

Answer 2

You can try to increase the Java heap size by specifying e.g.

java -Xmx1024M MyClass

on the command line (or what ever value will suit your document size).

Answer 3

Most likely you're not using SAX correctly, or your application isn't suited for stream processing.

The whole point of SAX is to avoid keeping the entire XML structure in memory, but that's only possible if you can process the XML in small chunks without keeping much context, and if the result of the processing either is much smaller than the processed XML (so that it does not use too much memory either) or can itself be passed on to a recipient or written to disk continuously.

Edit: It's also possible that you simply have a memory leak, i.e. you're holding on to data that you don't need anymore , preventing it from getting garbage collected. If you use any Lists, Maps or Sets for processing the XML, make sure that anything you add to them while processing one chunk of XML is removed before you start the next chunk.

Answer 4

StAX for Java versions pre-6: http://stax.codehaus.org/

Answer 5

Say you have the following XML structure:

<?xml version="1.0"?>
<list>
  <item>
    <name>Alpha</name>
    <age>10</age>
  </item>
  <item>
    <name>Beta</name>
    <age>20</age>
  </item>
  <!-- many many items -->
</list>

And you want to get all the <item>s

public class Item { String name; String age; }

Your SAX handler will look like this

public class MyHandler extends DefaultHandler
{
 Item current=null;
 StringBuilder content=null;
 @Override
    public void startElement(String uri, String localName, String name,
                    Attributes attributes) throws SAXException {
            if(     name.equals("item")
                  {
                  current= new Item();
                  }
            else if(name.equals("name") || name.equals("age"))
                  {
                  content= new StringBuilder();
                  }
            }

     @Override
    public void endElement(String uri, String localName, String name)
                    throws SAXException
            {
            if(name.equals("item"))
                    {
                   //DO SOMETHING WITH current
                    System.out.println(current);
                    current=null;
                    }
             else if(name.equals("name"))
                    {
                    current.name= content.toString();
                    }
             else if(name.equals("age"))
                    {
                    current.age= content.toString();
                    }
             content=null;
             }

    @Override
    public void characters(char[] ch, int start, int length)
                    throws SAXException {
            if(content!=null)
                    {
                    content.append(ch,start,length);
                    }
            }

}

As you can see, the 'content' is only memorized between the 'age' and 'name' tags.

Answer 6

Take a look at Apache Digester.

Here is a small tutorial