Updating from Java VM 5 to 6 gave a really big increase of memory consumption

Question

Hi, I've be working with a Java application run through the command-line. It deals with XML files, specially the dblp.xml database which has more than 400MB.

I was using JVM 5 and my app needed sort of 600-700MB of memory to processe the dblp.xml. After updating to JVM 6, it starting needing more than 1gb of memory (something I don't have), although it runs a bit faster.

I'm pretty sure of the memory consumption difference, because I've already tested both again and again in this same computer. Resulting in the same difference of memory consumption.

I didn't set any special parameters, just -Xmx800M or -Xmx1000M. Running with Ubuntu Hardy Heron on a dual core 1.7ghz, with 1,5gb of memory Using only the top/ps commands to measure

Any one have an idea why this occurs? I really wanted to use JVM 6, because in my production server it is the JVM in use, and I'm not quite able to change easily.

Thanks

Answer 1

My advice would be to use a tool like JProfiler to try to get a handle on exactly where your memory consumption increased. If you can't spend on JProfiler, check out this list of open source java profiling tools.

Guessing at the cause of performance problems is a bad idea, because you are almost always wrong in your guess, and you waste time optimizing the wrong piece of the system. Once you have some objective data, you may be able to come up with an optimization reducing your memory consumption.

If it is true that the newer JVM is trading memory for speed (by caching) then there is likely a jvm arg you can use to stop this behavior. Best bet is to check out the release notes for Java 6 and see if any such feature is mentioned.

Answer 2

sounds like the newer JVM is trading memory space for speed (ie, caching or what not)?

Answer 3

Maybe you used a 32bit 1.5 JVM, and a 64bit 1.6 JVM? You can find out, whether it is a 32bit or a 64bit JVM with

java -version

A 64bit JVM needs roughly 30% more memory compared with a 32 bit JVM, since you need 8 bytes for each reference instead of 4 bytes.

Answer 4

Nothing in the command "java -version" tells me that it is the 64bit version (nor the 32 bit), is there any more specific way to check if its the 64bit version (although I think it's not). But I got it both from "apt-get".

I just tried running the application with -d32 argument (to force 32 bits, data model), it seems to be that he uses a little bit less memory, but It stills uses way more that it did with jvm 5

JVM 6:

/usr/lib/jvm/java-6-sun/jre/bin/java -version java version "1.6.0_07" Java(TM) SE Runtime Environment (build 1.6.0_07-b06) Java HotSpot(TM) Client VM (build 10.0-b23, mixed mode, sharing)

(on my server machine) java -version java version "1.6.0" Java(TM) SE Runtime Environment (build 1.6.0-b105) Java HotSpot(TM) Client VM (build 1.6.0-b105, mixed mode)

JVM 5:

java -version java version "1.5.0_16" Java(TM) 2 Runtime Environment, Standard Edition (build 1.5.0_16-b02) Java HotSpot(TM) Client VM (build 1.5.0_16-b02, mixed mode, sharing)

Thanks for the anwsers

Answer 5

Using only the top/ps commands to measure

You cannot measure the memory consumption of a JVM with top or ps.

A JVM reserves memory rather greedily. Sometimes it does this even without trying to free the already reserved memory by GCing. (You could try to specify the -Xms parameter to change this. Read this to learn (much) more.)

top and ps only show the amount of currently reserved memory, not the memory actually used by your Java app!

If you want to measure the used memory correctly, you have to measure from Java itself. A brute force approach would be:

System.gc();
Runtime runtime = Runtime.getRuntime();
long memUsedInBytes = runtime.totalMemory() - runtime.freeMemory();

(Please just for debugging purposes and never in production code! System.gc() is evil.)

But I guess you run into OutOfMemoryErrors with -Xmx800M and 1.6, while 1.5 works fine. If this is the case, this answer is only a sidenote, because you do have a difference in memory consumption then.

Answer 6

What are you using to parse the xml file? There was a bug in the initial Java 6 library, that would cause this kind of behavior:

Bug 6536111

There is a workaround in the comments.

Answer 7

What JDK are you using? I would recommend you try out Sun's JDK (the server version), as the JRE (the client version) is not as memory efficient.

Answer 8

you can check with jconsole

x86 is 32 bit. If you really want to know where the memory is used get the Eclipse Memory Analyzer

Answer 9

Maybe your previous VM was running the client engine (i.e. "java -client") while the current one uses the server engine ("java -server"), which uses more memory in order to have much higher speeds. You can see this in the output of "java -version". The default engine to be used can be set in jvm.cfg (see http://forums.sun.com/thread.jspa?threadID=5185368&tstart=1314 for more details on that).