Out of memory error while creating huge XML

Question

Hi,

Caused by: java.lang.OutOfMemoryError: Java heap space
 at java.lang.AbstractStringBuilder.expandCapacity(Unknown Source)
 at java.lang.AbstractStringBuilder.append(Unknown Source)
 at java.lang.StringBuffer.append(Unknown Source)
 at java.io.StringWriter.write(Unknown Source)
 at com.ctc.wstx.sw.BufferingXmlWriter.flushBuffer(BufferingXmlWriter.java:1358)
 at com.ctc.wstx.sw.BufferingXmlWriter.flush(BufferingXmlWriter.java:224)
 at com.ctc.wstx.sw.BufferingXmlWriter.close(BufferingXmlWriter.java:198)
 at com.ctc.wstx.sw.BaseStreamWriter._finishDocument(BaseStreamWriter.java:1429)
 at com.ctc.wstx.sw.BaseStreamWriter.close(BaseStreamWriter.java:264)
 at org.codehaus.stax2.ri.Stax2EventWriterImpl.close(Stax2EventWriterImpl.java:178)
 at org.utils.JcoFunctionToXmlTransformer.transform(JcoFunctionToXmlTransformer.java:163

GOAL - To convert the data returned by SAP in form of JcoTable to XML.
PROBLEM - java.lang.OutOfMemoryError: Java heap space.

When the data is very huge, in cases where number of rows exceeds 25,000 getting the above error We are facing the same issue, even while using the Jco API method to convert to XML and also using the external custom code (using Stax API) to read node by node and stream as XML

Answer 1

Have you increased the VM options on Java when you run your app? For example:

java -Xmx1024m ...

to give it a gigabyte of heap space.

If this is within an application server or Web container you'll usually find such options buried in a config file or startup script.

Answer 2

You can increase the JVM's memory limit by saying:

java -Xmx512m ...

or

java -Xmx1024m ...

or whatever size you need.

Note that you may need to revise your algorithm to eg, serialize the XML directly to an OutputStream rather than first build a huge memory structure and then serialize that to the OutputStream. (This all depends on the details of what the code is doing.)

I had a similar case earlier this year where I was generating XML into a StringBuffer and then writing the StringBuffer into an HTTP response OutputStream. This worked fine until someone asked for 200Mb of XML! I quickly altered the code to generate the XML directly to the OutputStream, saving not only the memory, but CPU.

Answer 3

Dó you use substrings of large strings a lot?

attach with jvisualvm to see where your memory goes.

Answer 4

If you need to parse big XML files (and adding to the Java heap does not always work), you need a SAX parser which allows you to parse the XML stream instead of loading the whole DOM tree into memory.

Answer 5

java -Xmx512m ...

would help

Answer 6

Presumably you are writing the XML document out to a stream, in which case you don't need to hold the entire xml doc in memory as you can write it out to a stream as you construct it.