views:

1953

answers:

10

Does anyone have experience with using very large heaps 12 gb or higher in Java?

  • Does the GC make the program ununable?
  • What GC params do you use?
  • Which jvm sun or bea would be better suited for this?
  • Which platform linux or windows performs better under such conditions?
  • In the case of windows is there any performance difference to be had between 64 bit vista and xp under such high memory loads?
+10  A: 

We have an application that we allocate 12-16 Gb for but it really only reaches 8-10 during normal operation. We use the Sun JVM (tried IBMs and it was a bit of a disaster but that just might have been ignorance on our part...I have friends that swear by it--that work at IBM). As long as you give your app breathing room, the JVM can handle large heap sizes with not too much GC. Plenty of 'extra' memory is key.
Linux is almost always more stable than Windows and when it is not stable it is a hell of a lot easier to figure out why. Solaris is rock solid as well and you get DTrace too :) With these kind of loads, why on earth would you be using Vista or XP? You are just asking for trouble. We don't do anything fancy with the GC params. We do set the minimum allocation to be equal to the maximum so it is not constantly trying to resize but that is it.

Ichorus
I would not say that Linux was more stable then Windows, however it is very possible that Sun test's it's JVM more on unit and linex then it does on windows.
Ian Ringrose
+2  A: 

I recommend also considering taking a heap dump and see where memory usage can be improved in your app and analyzing the dump in something such as Eclipse's MAT . There are a few articles on the MAT page on getting started in looking for memory leaks. You can use jmap to obtain the dump with something such as ...

jmap -heap:format=b pid
jlintz
...and how is this answering the actual question?
ddimitrov
because with a heap size that large, you should be looking to reduce the memory footprint as well as optimizing the JVM
jlintz
A: 

The max memory that XP can address is 4 gig(here). So you may not want to use XP for that(use a 64 bit os).

Milhous
Or use the 64 bit version of XP. ;)
Tyler Millican
This isn't a limitation of XP, it's a limitation of any 32-bit OS that doesn't use PAE.
TM
It's a limitation of all 32-bit OSs, even those that use PAE.
James
@james, If you are using a PAE you will see the entire 4GB, if you dont have PAE, then you will not see devices that are mapped to memory(graphics cards etc).
Milhous
+1  A: 

If you switch to 64-bit you will use more memory. Pointers become 8 bytes instead of 4. If you are creating lots of objects this can be noticeable seeing as every object is a reference (pointer).

I have recently allocated 15GB of memory in Java using the Sun 1.6 JVM with no problems. Though it is all only allocated once. Not much more memory is allocated or released after the initial amount. This was on a Linux but I imagine the Sun JVM will work just as well on 64-bit Windows.

cdv
A: 

sun has had an itanium 64-bit jvm for a while although itanium is not a popular destination. The solaris and linux 64-bit JVMs should be what you should be after.
Some questions

1) is your application stable ?
2) have you already tested the app in a 32 bit JVM ?
3) is it OK to run multiple JVMs on the same box ?

I would expect the 64-bit OS from windows to get stable in about a year or so but until then, solaris/linux might be better bet.

anjanb
A: 

an article from sun on java 6 can help you : http://java.sun.com/developer/technicalArticles/javase/troubleshoot/

anjanb
+7  A: 

12Gb should be no problem with a decent JVM implementation such as Sun's Hotspot. I would advice you to use the Concurrent Mark and Sweep colllector ( -XX:+UseConcMarkSweepGC) when using a SUN VM.Otherwies you may face long "stop the world" phases, were all threads are stopped during a GC.

The OS should not make a big difference for the GC performance.

You will need of course a 64 bit OS and a machine with enough physical RAM.

kohlerm
+1  A: 

here's an article on gc FROM one of Java Champions -- http://kirk.blog-city.com/is_your_concurrent_collector_failing_you.htm

Kirk, the author writes "Send me your GC logs

I'm currently interested in studying Sun JVM produced GC logs. Since these logs contain no business relevent information it should be ease concerns about protecting proriatary information. All I ask that with the log you mention the OS, complete version information for the JRE, and any heap/gc related command line switches that you have set. I'd also like to know if you are running Grails/Groovey, JRuby, Scala or something other than or along side Java. The best setting is -Xloggc:. Please be aware that this log does not roll over when it reaches your OS size limit. If I find anything interesting I'll be happy to give you a very quick synopsis in return. "

anjanb
Please update the link to http://kirk.blog-city.com/is_your_concurrent_collector_failing_you.htm
Claes Mogren
A: 

You should try running visualgc against your app. It´s a heap visualization tool that´s part of the jvmstat download at http://java.sun.com/performance/jvmstat/

It is a lot easier than reading GC logs.

It quickly helps you understand how the parts (generations) of the heap are working. While your total heap may be 10GB, the various parts of the heap will be much smaller. GCs in the Eden portion of the heap are relatively cheap, while full GCs in the old generation are expensive. Sizing your heap so that that the Eden is large and the old generation is hardly ever touched is a good strategy. This may result in a very large overall heap, but what the heck, if the JVM never touches the page, it´s just a virtual page, and doesn´t have to take up RAM.

tpgould
+1  A: 

A couple of years ago, I compared JRockit and the Sun JVM for a 12G heap. JRockit won, and Linux hugepages support made our test run 20% faster. YMMV as our test was very processor/memory intensive and was primarily single-threaded.

ShabbyDoo
What Java version was that, and would you have time to do it again today? The numbers would be very intersting.
Thorbjørn Ravn Andersen
I'm not consulting for the same company anymore, so I don't even have the environment to try this out.It was a JDK1.5 JRockit, IIRC.
ShabbyDoo