I can see the image but unable to read it properly as to what is on X and Y axis. I am assuming the frequency of GC is increasing after sometime of JVM running. This could be because you getting more load at that time. If you have switched on GC logs then you can check if JVM have changed the GC strategy in middle.
A:
Bhushan
2009-04-21 10:27:47
Click the image (it's a link) to see the full-size version with readable labels.
Hank Gay
2009-04-21 10:38:35
+5
A:
The problem seems to be that the system can't recover sufficient memory even after a full garbage collection. This also has the negative effect of promoting objects to tenured and survivor spaces that don't really belong there.
The size of the old generation should settle down over time, most objects are (or at least should be) very short lived. If you find that the old generation keeps growing then there may be a resource leak somewhere. From your graph it seems like some allocation is tipping the system over the edge and into a gc death spiral.
Try to get the an idea of what objects are being created, and, more importantly, how long they are hanging around. This will give you the best idea of where you should be looking.
CurtainDog
2009-04-21 11:03:09
The only problem I have with that analysis is the sudden doubling in size of the heap (~12GB to ~24GB) that happens in a matter of seconds.RE: allocation - What's the best tool for that? I'm REALLY interested in the allocations that happen during that humongous spike, but I haven't found a good way to get at that particular info.Thanks again.
Hank Gay
2009-04-21 11:17:14
The Java VisualVM tool is included in JDK 6 update 7 which should make getting this information easier but I haven't used it myself.
CurtainDog
2009-04-21 11:45:56
Try and get a heap dump (jmap -dump:format=b,file=/some.file {java pid}) before and after the problem starts. Then you can use jhat to compare the two heaps and report on the difference. N.B. you'll need a machine with a lot of CPU and RAM to run JHat on such large heap dumps. Also, make sure you output the jmap dump files to physical disk, not swap-mounted disk (like /tmp) as you'll probably run out of memory
Chris May
2009-04-22 13:31:16