RSS/VSS are growing unstopped until all memory & swap on machine ends

Question

Hi,

We have a weblogic 9.2 server with Java1.5.0.16 on RHEL5.3 that we deploy on it a web service and an Alfresco content management system.

We were running it fine for ~3 years on HP-UX i11.23 and a month ago we moved to Linux RH5.3 and from time to time (it happened 3 times) we noticed that the process is starting to use more and more memory until all the memory and swap on the machine ends.

The process still works fine and all the log files looks normal (as if nothing happened) including GC log.

Glance for process ID 25450:

B0000A Glance C.04.70.000 06:54:05 supra2 x86_64 Current Avg High
CPU Util SU | 2% 2% 2%
Disk Util D D | 97% 97% 97%
Mem Util U U | 98% 98% 98%
Swap Util U U | 60% 60% 60%
Resources PID: 25450, java PPID: 25394 euid: 664 User:afspr04
CPU Usage (util): 5.40 Total RSS : 40.6gb
User CPU : 3.60 Text VSS : 56kb
System CPU : 1.80 Data VSS : 66.1gb
Priority : 15 Stack VSS : 2.0mb
Nice Value : 0 Total VSS : 66.5gb
Blocked On : SLEEP
Major Faults : 235
Minor Faults : 164
Processor : 1
Argv1: weblogic.Server
Cmd : /opt/java1.5.0_16/bin/java -Dweblogic.Name=dmcmsserver -Doracle.net.tns_admin=/etc -server -javaagent:/opt/MercuryDiagn
ostics/JavaAgent/DiagnosticsAgent/lib/probeagent.jar -Dprobe.id=supra2_afspr04_dmcms_ear_p4 -Dprobe.group=CMS_SERVER -D
points.file.name=/opt/MercuryDiagnostics/JavaAgent/DiagnosticsAgent/etc/supra2_afspr04_dmcms_ear_p4 -Dcom.wily.introsco
pe.agent.agentName=DMCMS -Xms7g -Xmx7g -XX:PermSize=256m -XX:MaxPermSize=256m -XX:NewSize=1792m -XX:MaxNewSize=1792m -X
X:SurvivorRatio=4 -XX:TargetSurvivorRatio=90 -XX:+DisableExplicitGC -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -Xnoclassg
c -Xloggc:logs/gc.log -Doracle.net.tns_admin=/etc -Dweblogic.Stderr=/app/afspr04/dmcms_ear_p4/dmcmsdomain/logs/online.l
og -Dweblogic.Stdout=/app/afspr04/dmcms_ear_p4/dmcmsdomain/logs/online.log -Damdocs.system.home=/app/afspr04/dmcms_ear_
p4/properties/jesi -Damdocs.messageHandling.home=/app/afspr04/dmcms_ear_p4/properties/jesi -Djesi.config.loader=amdocs.
ecommerce.esi.utils.config.InterfaceConfigXPathLoader -Damdocs.uams.config.resource=config/mvc/ldap ...

pmap shows the big allocation as anonymous pmap (sorted by the big once):

25450: /opt/java1.5.0_16/bin/java -Dweblogic.Name=dmcmsserver -Doracle.net.tns_admin=/etc -server -javaagent:/opt/MercuryDiagnostics/JavaAgent/DiagnosticsAgent/lib/probeagent.jar -Dprobe.id=supra2_afspr04_dmcms_ear_p4 -Dprobe.group=CMS_SERVER -Dpoints.file.name=/opt/MercuryDiagnostics/JavaAgent/DiagnosticsAgent/etc/supra2_afspr04_dmcms_ear_p4 -Dcom.wily.introscope.agent.agentName=DMCMS -Xms7g -Xmx7g -XX:PermSize=256m -XX:MaxPermSize=256m -XX:NewSize=1792m -XX:MaxNewSize=1792m -XX:SurvivorRatio=4 -XX:TargetSurvivo
00002ab0f8000000    10518548    rwx--   [anon]
00002ab798009000    8388612 rwx--   [anon]
000000005fcce000    8038976 rwx--   [anon]
00002aac7aab0000    7602176 rwx--   [anon]
00002aaf74000000    5259284 rwx--   [anon]
00002ab688000000    4194308 rwx--   [anon]
00002aae4b930000    1684124 rwx--   [anon]
00002aab80000000    1314836 rwx--   [anon]
00002aab20000000    655376  rwx--   [anon]
00002aac28000000    532488  rwx--   [anon]
00002aac50000000    524292  rwx--   [anon]
00002aaaec000000    327696  rwx--   [anon]
00002aaad8000000    131088  rwx--   [anon]
00002ab658000000    131060  rwx--   [anon]
00002ab0dc000000    131044  rwx--   [anon]
00002aaacc2f5000    114708  rwx--   [anon]
...
total 69733292K 

Have anyone encountered something similar?

Thanks, Oz

Answer 1

What is the CPU/RAM of the server you are using? You should consult the RHEL compatibility matrix for WLS 9.2 and make sure your JDK/CPU configuration is a supported combination. Also, since you might want to consider JRockit as your JVM if that's an option for you. Finally, also try lowering the max heap space (-Xmx and -Xms) and see if the server is more stable.