Java has a 39G core dump...

Question

I'm running a weblogic server on solarix x86 - 64bit with the command line:

-Xrs -Xms4096m -Xmx4096m -XX:MaxPermSize=256m -da ...

so the max heap size should be 4G, however after a night, it crashed and generated a 39G core:

-bash-3.00$ ls -l core
-rw-------   1 user group     39017429722 Sep 27 19:47 core

I used pmap to dump the core content:

$ pmap core
core 'core' of 21092:   /opt/middleware/jdk1.6.0_21/bin/amd64/java -Xrs -Xms
0000000000400000         52K r-x--  /opt/middleware/jdk1.6.0_21/bin/amd64/java
000000000041C000          4K rw---  /opt/middleware/jdk1.6.0_21/bin/amd64/java
000000000041D000    2226208K rw---
0000000088225000    2097152K rw---
0000000108225000    4194304K rw---
0000000208225000    8388608K rw---
0000000408225000   16777216K rw---    [ heap ]
FFFFFD7EDF610000        512K rwx--
FFFFFD7EDF77A000         96K rw---    [ stack tid=147 ]
FFFFFD7EDF87B000         96K rw---    [ stack tid=146 ]
FFFFFD7EDF97C000         96K rw---    [ stack tid=145 ]
FFFFFD7EDFA7D000         96K rw---    [ stack tid=144 ]
.....
     total     38102164K

As you can see there is a 16G heap over there...why this happens? java memory leak?

jmap dump:

-bash-3.00$ /opt/middleware/jdk1.6.0_21/bin/jmap -heap /opt/middleware/jdk1.6.0_21/bin/java -d64 core
Attaching to core core from executable /opt/middleware/jdk1.6.0_21/bin/java, please wait...
Debugger attached successfully.
Server compiler detected.
JVM version is 17.0-b16

using thread-local object allocation.
Parallel GC with 4 thread(s)

Heap Configuration:
   MinHeapFreeRatio = 40
   MaxHeapFreeRatio = 70
   MaxHeapSize      = 4294967296 (4096.0MB)
   NewSize          = 1310720 (1.25MB)
   MaxNewSize       = 17592186044415 MB
   OldSize          = 5439488 (5.1875MB)
   NewRatio         = 2
   SurvivorRatio    = 8
   PermSize         = 21757952 (20.75MB)
   MaxPermSize      = 268435456 (256.0MB)

Heap Usage:
PS Young Generation
Eden Space:
   capacity = 1226899456 (1170.0625MB)
   used     = 396823336 (378.44022369384766MB)
   free     = 830076120 (791.6222763061523MB)
   32.34359050852819% used
From Space:
   capacity = 104726528 (99.875MB)
   used     = 2949120 (2.8125MB)
   free     = 101777408 (97.0625MB)
   2.816020025031289% used
To Space:
   capacity = 100728832 (96.0625MB)
   used     = 0 (0.0MB)
   free     = 100728832 (96.0625MB)
   0.0% used
PS Old Generation
   capacity = 2864709632 (2732.0MB)
   used     = 90771752 (86.56668853759766MB)
   free     = 2773937880 (2645.4333114624023MB)
   3.1686196390043064% used
PS Perm Generation
   capacity = 140509184 (134.0MB)
   used     = 139181296 (132.73362731933594MB)
   free     = 1327888 (1.2663726806640625MB)
   99.05494576069846% used

Additional information: so files loaded in this core:

-bash-3.00$ grep so p.txt|sort -k 4
FFFFFD7FFF3B2000        228K r-x--  /lib/amd64/ld.so.1
FFFFFD7FFF3FB000          8K rwx--  /lib/amd64/ld.so.1
FFFFFD7FFF3FD000          8K rwx--  /lib/amd64/ld.so.1
FFFFFD7EE8100000         32K r-x--  /lib/amd64/libaio.so.1
FFFFFD7EE8118000          4K rw---  /lib/amd64/libaio.so.1
FFFFFD7EE8119000          4K rw---  /lib/amd64/libaio.so.1
FFFFFD7FFF1F0000       1252K r-x--  /lib/amd64/libc.so.1
FFFFFD7FFF339000         36K rw---  /lib/amd64/libc.so.1
FFFFFD7FFF342000         16K rw---  /lib/amd64/libc.so.1
FFFFFD7FFF350000          4K r-x--  /lib/amd64/libdl.so.1
FFFFFD7FFF361000          4K rw---  /lib/amd64/libdl.so.1
FFFFFD7FFE390000          8K r-x--  /lib/amd64/libdoor.so.1
FFFFFD7FFE3A2000          4K rw---  /lib/amd64/libdoor.so.1
FFFFFD7FFE1E0000         28K r-x--  /lib/amd64/libgen.so.1
FFFFFD7FFE1F7000          4K rw---  /lib/amd64/libgen.so.1
FFFFFD7FFE3E0000         16K r-x--  /lib/amd64/libm.so.1
FFFFFD7FFE3F3000          4K rw---  /lib/amd64/libm.so.1
FFFFFD7FFE250000        348K r-x--  /lib/amd64/libm.so.2
FFFFFD7FFE2B6000         24K rw---  /lib/amd64/libm.so.2
FFFFFD7FFE1C0000         48K r-x--  /lib/amd64/libmd.so.1
FFFFFD7FFE1DC000          4K rw---  /lib/amd64/libmd.so.1
FFFFFD7FFE1A0000         16K r-x--  /lib/amd64/libmp.so.2
FFFFFD7FFE1B4000          4K rw---  /lib/amd64/libmp.so.2
FFFFFD7FFE2C0000        664K r-x--  /lib/amd64/libnsl.so.1
FFFFFD7FFE376000         16K rw---  /lib/amd64/libnsl.so.1
FFFFFD7FFE37A000         36K rw---  /lib/amd64/libnsl.so.1
FFFFFD7EE8120000         32K r-x--  /lib/amd64/librt.so.1
FFFFFD7EE8138000          4K rw---  /lib/amd64/librt.so.1
FFFFFD7FFE220000        100K r-x--  /lib/amd64/libscf.so.1
FFFFFD7FFE249000          4K rw---  /lib/amd64/libscf.so.1
FFFFFD7FFF190000         60K r-x--  /lib/amd64/libsocket.so.1
FFFFFD7FFF1AF000          4K rw---  /lib/amd64/libsocket.so.1
FFFFFD7FFF3A0000         20K r-x--  /lib/amd64/libthread.so.1
FFFFFD7FFE200000         36K r-x--  /lib/amd64/libuutil.so.1
FFFFFD7FFE219000          4K rw---  /lib/amd64/libuutil.so.1
FFFFFD7FFF370000         36K r-x--  /opt/middleware/jdk1.6.0_21/jre/lib/amd64/jli/libjli.so
FFFFFD7FFF388000          8K rw---  /opt/middleware/jdk1.6.0_21/jre/lib/amd64/jli/libjli.so
FFFFFD7EE80D0000         64K r-x--  /opt/middleware/jdk1.6.0_21/jre/lib/amd64/libj2pkcs11.so
FFFFFD7EE80EF000          4K rw---  /opt/middleware/jdk1.6.0_21/jre/lib/amd64/libj2pkcs11.so
FFFFFD7FFDFE0000        188K r-x--  /opt/middleware/jdk1.6.0_21/jre/lib/amd64/libjava.so
FFFFFD7FFE01E000         12K rw---  /opt/middleware/jdk1.6.0_21/jre/lib/amd64/libjava.so
FFFFFD7EE7DF0000         28K r-x--  /opt/middleware/jdk1.6.0_21/jre/lib/amd64/libmanagement.so
FFFFFD7EE7E06000          4K rw---  /opt/middleware/jdk1.6.0_21/jre/lib/amd64/libmanagement.so
FFFFFD7EE82A0000         72K r-x--  /opt/middleware/jdk1.6.0_21/jre/lib/amd64/libnet.so
FFFFFD7EE82C1000          8K rw---  /opt/middleware/jdk1.6.0_21/jre/lib/amd64/libnet.so
FFFFFD7EE8140000         32K r-x--  /opt/middleware/jdk1.6.0_21/jre/lib/amd64/libnio.so
FFFFFD7EE8157000          4K rw---  /opt/middleware/jdk1.6.0_21/jre/lib/amd64/libnio.so
FFFFFD7FFE030000         68K r-x--  /opt/middleware/jdk1.6.0_21/jre/lib/amd64/libverify.so
FFFFFD7FFE050000          8K rw---  /opt/middleware/jdk1.6.0_21/jre/lib/amd64/libverify.so
FFFFFD7FFDF70000         64K r-x--  /opt/middleware/jdk1.6.0_21/jre/lib/amd64/libzip.so
FFFFFD7FFDF8F000         12K rw---  /opt/middleware/jdk1.6.0_21/jre/lib/amd64/libzip.so
FFFFFD7FFDFC0000         40K r-x--  /opt/middleware/jdk1.6.0_21/jre/lib/amd64/native_threads/libhpi.so
FFFFFD7FFDFD9000          4K rw---  /opt/middleware/jdk1.6.0_21/jre/lib/amd64/native_threads/libhpi.so
FFFFFD7FFDFDA000         20K rw---  /opt/middleware/jdk1.6.0_21/jre/lib/amd64/native_threads/libhpi.so
FFFFFD7FFE400000      12932K r-x--  /opt/middleware/jdk1.6.0_21/jre/lib/amd64/server/libjvm.so
FFFFFD7FFF0B2000        628K rw---  /opt/middleware/jdk1.6.0_21/jre/lib/amd64/server/libjvm.so
FFFFFD7FFF14F000        112K rw---  /opt/middleware/jdk1.6.0_21/jre/lib/amd64/server/libjvm.so
FFFFFD7FFE3B0000         60K r-x--  /usr/lib/amd64/libCrun.so.1
FFFFFD7FFE3CE000          8K rw---  /usr/lib/amd64/libCrun.so.1
FFFFFD7FFE3D0000         24K rw---  /usr/lib/amd64/libCrun.so.1
FFFFFD7EE8070000         28K r-x--  /usr/lib/amd64/libcryptoutil.so.1
FFFFFD7EE8087000          8K rw---  /usr/lib/amd64/libcryptoutil.so.1
FFFFFD7EE8090000        136K r-x--  /usr/lib/amd64/libpkcs11.so.1
FFFFFD7EE80C2000          4K rw---  /usr/lib/amd64/libpkcs11.so.1
FFFFFD7EE80C3000          4K rw---  /usr/lib/amd64/libpkcs11.so.1
FFFFFD7FFF1C0000          4K r-x--  /usr/lib/amd64/libsched.so.1
FFFFFD7EE8010000        304K r-x--  /usr/lib/security/amd64/pkcs11_softtoken_extra.so.1
FFFFFD7EE806C000         12K rw---  /usr/lib/security/amd64/pkcs11_softtoken_extra.so.1

Answer 1

-Xmx4096m is the size of the java Heap. The core dump is about the entire java Virtual Machine address space process.

Depending on the operating system, you can get a virtual addressing space bigger then expected. So the dump could be greater.

For instance, WindowsXP has a 4GB address space for every process, even when you run on 256MB of RAM.

I think the core dump size depends on the Java VM + O.S. implementation. For instance the Solaris and linux core dumps have nothing in common at all.

UPDATE: Two options: Are you using native libraries (i.e. non standard stuff)? If so disable the native extension(s) and do some tests. Also disabling Java NIO could help.

If you are using plain vanilla WebLogic, try out your support: 64bit it is not (yet) so common in big deployments, and you seems to have a very big one (you have much RAM then I even seen :)