ansaurus

Question

Answer 1

+2 A:

Everytime you use a DOM to parse a XML file, you'll load entire file into memory and DOM infrastructure will use about same size to handle it, so it'll consume about twice memory than your file size.

You'll need to use SAX, an event based parser. While this can be hard to understand it first time, it's a very memory effective, as it just keeps in memory current parsing node.

Seems Java have some SAX implementations, like StAX, I hope it helps.

Rubens Farias 2009-11-30 09:22:52

Hi Rubens, I am using JDOM for parsing the large XML and it uses SAX parser internally. My parsing code is : SAXBuilder builder = new SAXBuilder();Document doc = builder.build(inputResource);Element elem = doc.getRootElement();

Amit 2009-11-30 09:43:23

since your DOM parser uses SAX, you should read your XML sequentially, and to avoid to use `..`, `//` and stuff

Rubens Farias 2009-11-30 09:49:23

Answer 2

A:

Are you sure there are no recursive array copy somewhere, left there by mistake? Perhaps in different threads?

lorenzog 2009-11-30 09:23:58

Hi Lorenzog, I am not using any thread, for this purpose and no array copy. I am using JDOM for parsing the XML file and I guess it uses ArrayList for its implementation. Can that be an issue? Is there any possibility of memory leak?

Amit 2009-11-30 09:47:32

Answer 3

A:

I'll second that point about the file and the DOM taking up a great deal of memory. I also wonder when I see this:

ERROR [JobRunShell]: Job updateVendorData.quoteUpdate threw an unhandled Exception:  
    java.lang.OutOfMemoryError: Java heap space
    at java.util.Arrays.copyOfRange(Arrays.java:3210)

What's that copying doing? I wonder if there's something else bad going on in your code.

If you've gotten this far, it suggests that you've read the file and the DOM successfully and you're starting to write to the database. The file memory should already be reclaimed.

I'd suggest looking at memory using VisualGC so you can see what's going on.

duffymo 2009-11-30 11:02:52

That copying is the internals of StringBuffer.

BalusC 2009-11-30 11:21:19

Hi duffy, thanks for your reply. Can JDOM internally implement this copying ? Always I have enabled profiling for the Application and there are 2 classes which exist even after GC has run, they are org.postgresql.jdbc4.Jdbc4PrepareStatement and org.postgresql.jdbc4.Jdbc4ResultSet. Can these be cause of memory leak in the Application ?

Amit 2009-11-30 11:59:10

Could be. Can't tell without code, but if you aren't closing these properly you could be leaking resources that will bring you grief.

duffymo 2009-11-30 23:35:41

With the best of my abilities, I have taken care that I close all the connections which I have opened but the problem still exists. Is there any way you can help me out with this

Amit 2009-12-01 06:43:30

Not without more info.

duffymo 2009-12-01 10:58:07

Tell me how can i can provide you more info..

Amit 2009-12-01 11:14:11

Answer 4

+1 A:

Parsing XML is an fairly expensive task. The average DOM parser would already need at least five times of the memory space as the XML document big is. You should take this fact into account as well. To ensure that there is no memory leak somewhere else which caused the memory shortage for the XML parser, you really need to run a profiler. Give it all more memory, double the available memory and profile it. When you've nailed the cause down and fixed the leak, then you can just fall back to the "default" memory and retest. Or if there is really no means of any leak, then just give it all a bit more memory than default so that it all suits.

You can also consider to use a more memory efficient XML parser instead, for example VTD-XML (homepage here, benchmarks here).

BalusC 2009-11-30 11:31:43

Answer 5

A:

Have you tried setting the max heap size bigger to see if the problem still occurs then? There may not even be a leak at all. It might just be that the default heap size (64m on Windows I think) is insufficient for this particular process.

I find that I almost always need to give any application I'm running Tomcat more heap and perm gen space than the defaults or I'll run into out of memory problems. If you need help adjusting the memory settings take a look at this question.

Jason Gritman 2009-11-30 11:54:30

Hi Jason, thanks for your reply. I increased my heap size and the application worked just fine. I have set the profiling for my application and after the process was completed, could find 2 kinds of live allocated objects not being deallocated by garbage collector, they are org.postgresql.jdbc4.Jdbc4PrepareStatement and org.postgresql.jdbc4.Jdbc4ResultSet. Can these be cause of memory leak in the Application ?

Amit 2009-11-30 12:15:34

That might be an indication you aren't closing you're JDBC objects correctly. Are you doing the JDBC calls yourself or are you using a framework (like Spring) to wrap your JDBC calls? If you are calling JDBC directly, make sure you call the close() method on any ResultSet, Statement, PreparedStatement, and Connection objects in a finally block when you are finished using them.

Jason Gritman 2009-11-30 13:10:48

With the best of my abilities, I have taken care that I close all the connections which I have opened but the problem still exists. Is there any way you can help me out with this

Amit 2009-12-01 09:03:22

Answer 6

A:

You could run your application with: -XX:+HeapDumpOnOutOfMemoryError. This will cause the JVM to produce a heap dump when it runs out of memory. You can the use something like: MAT or JHAT to see what objects are being held on to. I suggest using the eclipse memory analyzer tool (MAT) on the generated heap dump as it is fairly straightforward to use: http://www.eclipse.org/mat/

Of course you will need to have some idea as to what objects may be hanging around in order for this to be useful. DOM objects? Resources from previous loads of xml documents? Database connections? MAT will allow you to trace the references back to a root object from some object that you suspect should have been garbage collected.

mlaw 2009-11-30 16:02:12

ansaurus

tags:

views:

answers:

Java out of memory Exception

related questions