Hello,
I am working with a java application deployed on an HP server via WebLogic 10.3.
Version info:
WebLogic Version 10.3
OS Version B.11.23
$ java -version
java version "1.6.0.03"
Java(TM) SE Runtime Environment (build 1.6.0.03-jinteg_05_feb_2009_11_19-b00)
Java HotSpot(TM) Server VM (build 11.0-b16-jre1.6.0.03-rc2, mixed mode)
I am getting this java.lang.OutOfMemoryError and the weblogic managed server is shutting down. It's always on the compiler thread. The error message says to check swap and MAXDSIZ. I looked at both and they seemed ok (see below.) I've gone through the forums and looked at various things but none seem to fully apply. I've changed the heap and stack both up and down with no luck. Also, I am pretty sure this is the first 1.6 port of the application on HP.
# kctune -a -v -q maxdsiz
Tunable maxdsiz
Description Maximum size of the data segment of a 32-bit process (bytes)
Module vm
Current Value 3221225472
Value at Next Boot 3221225472
Value at Last Boot 3221225472
Default Value 1073741824
Constraints maxdsiz GT 262144
maxdsiz LT 4294963200
Can Change Immediately or at Next Boot
# swapinfo -tam
Mb Mb Mb PCT START/ Mb
TYPE AVAIL USED FREE USED LIMIT RESERVE PRI NAME
dev 4096 261 3835 6% 0 - 1 /dev/vg00/lvol2
dev 4096 263 3833 6% 0 - 1 /dev/vg00/lvol9
reserve - 1195 -1195
memory 4085 1598 2487 39%
total 12277 3317 8960 27% - 0 -
I get a core and a hs.log file. This question seems similar but not sire of the resolution.
http://stackoverflow.com/questions/1102544/jvm-errors-after-updating-to-java-6-update-14
I am posting the log file. None of the Heap spaces show near 100%. I am not used to looking at these and was hoping someone could help guide me to a solution. I see physical memory goes low but not 100%.
#
# An unexpected error has been detected by Java Runtime Environment:
#
# java.lang.OutOfMemoryError: requested 32756 bytes for ChunkPool::allocate. Out of swap space?
Possible causes:
- not enough swap space left, or
- kernel parameter MAXDSIZ is very small.
#
# Internal Error (allocation.cpp:128), pid=11593, tid=9
# Error: ChunkPool::allocate
#
# Java VM: Java HotSpot(TM) Server VM (11.0-b16-jre1.6.0.03-rc2 mixed mode hp-ux-ia64)
# Please report this error to HP customer support.
#
T H R E A D
Current thread (0059e400): JavaThread "CompilerThread0" daemon [_thread_in_native, id=9, lwp_id=3000034, stack(21e01000,22001
000)]
Stack: 21e01000,22001000
Current CompileTask:
C3:154% ! weblogic.timers.internal.TimerThread$Thread.run()V @ 0 (196 bytes)
P R O C E S S
Java Threads: (current thread )
01156000 JavaThread "STANDBY ExecuteThread: '4' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon [_thread_blocke
d, id=58, lwp_id=3000185, stack(1fa01000,1fa41000)]
04fa1c00 JavaThread "STANDBY ExecuteThread: '3' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon [_thread_blocke
d, id=57, lwp_id=3000183, stack(1fac1000,1fb01000)]
04bb1600 JavaThread "ACTIVE ExecuteThread: '2' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon [_thread_blocked
, id=56, lwp_id=3000182, stack(1fb81000,1fbc1000)]
04f81200 JavaThread "DynamicSSLListenThreadDefaultSecure" daemon [_thread_in_native, id=55, lwp_id=3000181, stack(1fc41000
,1fc81000)]
043d6e00 JavaThread "DynamicListenThreadDefault" daemon [_thread_in_native, id=54, lwp_id=3000180, stack(1fd01000,1fd41000
)]
043d3400 JavaThread "weblogic.GCMonitor" daemon _thread_blocked, id=53, lwp_id=3000178, stack(1fdc1000,1fe01000)
03167a00 JavaThread "Thread-14" daemon _thread_in_native, id=52, lwp_id=3000177, stack(1fe81000,1fec1000)
03e5a600 JavaThread "RecyclableThread - waiting" daemon _thread_blocked, id=51, lwp_id=3000155, stack(1ff41000,1ff81000)
03e56e00 JavaThread "RecyclableThread - waiting" daemon _thread_blocked, id=50, lwp_id=3000154, stack(20001000,20041000)
03e45000 JavaThread "RecyclableThread - waiting" daemon _thread_blocked, id=49, lwp_id=3000153, stack(200c1000,20101000)
03e2e200 JavaThread "RecyclableThread - waiting" daemon _thread_blocked, id=48, lwp_id=3000152, stack(20181000,201c1000)
03e29000 JavaThread "RecyclableThread - waiting" daemon _thread_blocked, id=47, lwp_id=3000151, stack(20241000,20281000)
03e27a00 JavaThread "RecyclableThread - waiting" daemon _thread_blocked, id=46, lwp_id=3000150, stack(20301000,20341000)
03e15a00 JavaThread "RecyclableThread - waiting" daemon _thread_blocked, id=45, lwp_id=3000149, stack(203c1000,20401000)
0073de00 JavaThread "RecyclableThread - waiting" daemon _thread_blocked, id=44, lwp_id=3000148, stack(20481000,204c1000)
0073da00 JavaThread "RecyclableThread - waiting" daemon _thread_blocked, id=43, lwp_id=3000147, stack(20c01000,20c41000)
0073d600 JavaThread "RecyclableThread - waiting" daemon _thread_blocked, id=42, lwp_id=3000146, stack(20cc1000,20d01000)
03e11800 JavaThread "SimpleQuartzScheduler_QuartzSchedulerThread" [_thread_blocked, id=41, lwp_id=3000139, stack(20541000,20
581000)]
03e0f400 JavaThread "ThreadPoolManager" daemon _thread_blocked, id=39, lwp_id=3000137, stack(206c1000,20701000)
03e0d000 JavaThread "DataCacheManager" daemon _thread_blocked, id=38, lwp_id=3000136, stack(20781000,207c1000)
03e0b400 JavaThread "TriggerManager" daemon _thread_blocked, id=37, lwp_id=3000135, stack(20841000,20881000)
03e09c00 JavaThread "RequestEventManager" daemon _thread_blocked, id=36, lwp_id=3000134, stack(20901000,20941000)
034b9200 JavaThread "HyperServiceCacheManager" daemon _thread_blocked, id=35, lwp_id=3000133, stack(209c1000,20a01000)
034b7800 JavaThread "LogManager" daemon _thread_blocked, id=34, lwp_id=3000132, stack(20a81000,20ac1000)
034b7400 JavaThread "ServiceTaskManager" daemon _thread_blocked, id=33, lwp_id=3000131, stack(21141000,21181000)
0073e800 JavaThread "NXAHSProcessMgr::MasterThread" daemon _thread_blocked, id=32, lwp_id=3000130, stack(20b41000,20b81000)
03ae6800 JavaThread "EventBroker" daemon _thread_blocked, id=29, lwp_id=3000102, stack(20d81000,20dc1000)
03aaec00 JavaThread "I/O Monitor" daemon _thread_blocked, id=28, lwp_id=3000101, stack(20e41000,20e81000)
033bf600 JavaThread "Timer-2" daemon _thread_blocked, id=27, lwp_id=3000100, stack(20f01000,20f41000)
0312be00 JavaThread "Thread-11" daemon _thread_blocked, id=26, lwp_id=3000057, stack(20fc1000,21001000)
0079cc00 JavaThread "DoSManager" daemon _thread_blocked, id=25, lwp_id=3000056, stack(21081000,210c1000)
030ca400 JavaThread "VDE Transaction Processor Thread" daemon [_thread_blocked, id=23, lwp_id=3000054, stack(21201000,212410
00)]
03185c00 JavaThread "ExecuteThread: '2' for queue: 'weblogic.socket.Muxer'" daemon [_thread_blocked, id=22, lwp_id=3000053,
stack(212c1000,21301000)]
0312d200 JavaThread "ExecuteThread: '1' for queue: 'weblogic.socket.Muxer'" daemon [_thread_blocked, id=21, lwp_id=3000052,
stack(21381000,213c1000)]
030cba00 JavaThread "ExecuteThread: '0' for queue: 'weblogic.socket.Muxer'" daemon [_thread_in_native, id=20, lwp_id=3000051
, stack(21881000,218c1000)]
00a1c800 JavaThread "ACTIVE ExecuteThread: '1' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon [_thread_blocked
, id=19, lwp_id=3000050, stack(21941000,21981000)]
030ab600 JavaThread "Thread-7" daemon _thread_blocked, id=18, lwp_id=3000049, stack(21a01000,21a41000)
00689600 JavaThread "weblogic.timers.TimerThread" daemon _thread_blocked, id=17, lwp_id=3000048, stack(21ac1000,21b01000)
00683e00 JavaThread "weblogic.time.TimeEventGenerator" daemon [_thread_blocked, id=16, lwp_id=3000047, stack(21b81000,21bc10
00)]
00677800 JavaThread "ACTIVE ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon [_thread_blocked
, id=15, lwp_id=3000046, stack(22081000,220c1000)]
00677000 JavaThread "Timer-1" daemon _thread_blocked, id=14, lwp_id=3000045, stack(22281000,222c1000)
02094e00 JavaThread "Timer-0" daemon _thread_blocked, id=13, lwp_id=3000038, stack(22481000,224c1000)
005a9e00 JavaThread "Low Memory Detector" daemon _thread_blocked, id=12, lwp_id=3000037, stack(22681000,226c1000)
005a1400 JavaThread "CompilerThread1" daemon _thread_blocked, id=10, lwp_id=3000035, stack(21601000,21801000)
=>0059e400 JavaThread "CompilerThread0" daemon _thread_in_native, id=9, lwp_id=3000034, stack(21e01000,22001000)
0059c200 JavaThread "Signal Dispatcher" daemon _thread_blocked, id=8, lwp_id=3000033, stack(22881000,228c1000)
0057cc00 JavaThread "Finalizer" daemon _thread_blocked, id=7, lwp_id=3000032, stack(231c1000,23201000)
00535600 JavaThread "Reference Handler" daemon _thread_blocked, id=6, lwp_id=3000031, stack(23281000,232c1000)
00465800 JavaThread "main" _thread_blocked, id=1, lwp_id=-1, stack(7fdfe000,7ffff000)
Other Threads:
0051aa00 VMThread stack: 22381000,22401000 id=5, lwp_id=3000030
0048a800 WatcherThread stack: 22181000,22201000 id=11, lwp_id=3000036
VM state: not at safepoint (normal execution)
VM Mutex/Monitor currently owned by a thread: None
Heap
PSYoungGen total 308544K, used 132524K [5deb0000, 72790000, 73400000)
eden space 307328K, 43% used [5deb0000,6601b248,70ad0000)
from space 1216K, 0% used [72660000,72660000,72790000)
to space 13888K, 0% used [70c70000,70c70000,71a00000)
PSOldGen total 174784K, used 90916K [33400000, 3deb0000, 5deb0000)
object space 174784K, 52% used [33400000,38cc9350,3deb0000)
PSPermGen total 226304K, used 113097K [23400000, 31100000, 33400000)
object space 226304K, 49% used [23400000,2a272528,31100000)
Dynamic libraries:
/opt/java1.6/jre/bin/IA64N/java
text:0x00400000-0x0041b040 data:0x0041c000-0x0041c4f8
/usr/lib/hpux32/libpthread.so.1
text:0xc008d000-0xc011fcb0 data:0x777f2000-0x777f4508
/opt/java1.6/jre/bin/IA64N/../../lib/IA64N/jli/libjli.so
text:0xc1794000-0xc17b15e0 data:0x777f0000-0x777f1498
/usr/lib/hpux32/libdl.so.1
text:0xc1953000-0xc1956ed0 data:0x777ed000-0x777ed150
/usr/lib/hpux32/libc.so.1
text:0xc0120000-0xc039e300 data:0x777d9000-0x777ea0f8
/opt/java1.6/jre/lib/IA64N/server/libjvm.so
text:0xcb8f6000-0xcd783450 data:0x775ae000-0x777bace8
/usr/lib/hpux32/libm.so.1
text:0xc052f000-0xc0691c30 data:0x777cc000-0x777cf818
/usr/lib/hpux32/librt.so.1
text:0xc4ebd000-0xc4ec31f0 data:0x777d4000-0x777d41a8
/usr/lib/hpux32/libuca.so.1
text:0xc0c1c000-0xc0c23af0 data:0x777c9000-0x777c9150
/usr/lib/hpux32/libelf.so.1
text:0xc17b2000-0xc17d9fe0 data:0x777c8000-0x777c88c8
/usr/lib/hpux32/libunwind.so.1
text:0xc0bd3000-0xc0c1b910 data:0x777c7000-0x777c7628
/usr/lib/hpux32/libstd_v2.so.1
text:0xc09e2000-0xc0b82fc0 data:0x775a0000-0x775adc28
/usr/lib/hpux32/libCsup.so.1
text:0xc0b83000-0xc0bd2190 data:0x7759a000-0x7759ff20
/opt/java1.6/jre/lib/IA64N/native_threads/libhpi.so
text:0xc17da000-0xc17f0a60 data:0x777bc000-0x777bd6a0
/opt/java1.6/jre/lib/IA64N/libverify.so
text:0xc1957000-0xc1976b20 data:0x777bb000-0x777bbaf0
/opt/java1.6/jre/lib/IA64N/libjava.so
text:0xcb79e000-0xcb801070 data:0x77591000-0x77593208
/opt/java1.6/jre/lib/IA64N/libzip.so
text:0xc1977000-0xc19ab810 data:0x7758d000-0x7758e800
/opt/java1.6/jre/lib/IA64N/libmanagement.so
text:0xc19b4000-0xc19befa0 data:0x7758a000-0x7758a300
/opt/java1.6/jre/lib/IA64N/libnet.so
text:0xcd784000-0xcd7b2b30 data:0x77589000-0x77589908
/usr/lib/hpux32/libnss_files.so.1
text:0xc0518000-0xc052ea70 data:0x77586000-0x775868e0
/opt/java1.6/jre/lib/IA64N/libnio.so
text:0xc17f9000-0xc1808160 data:0x77585000-0x77585588
/d1/app/bea/wlserver_10.3/server/native/hpux11/IPF32/libwlfileio2.so
text:0xc18ab000-0xc18afb50 data:0x77582000-0x77582218
/d1/app/bea/wlserver_10.3/server/native/hpux11/IPF32/libmuxer.so
text:0xc5f0e000-0xc5f14f10 data:0x77581000-0x775818a0
/usr/lib/hpux32/libnss_nisplus.so.1
text:0xc7756000-0xc776fb40 data:0x7753e000-0x7753f580
/usr/lib/hpux32/libsec.so.1
text:0xc04d5000-0xc0517f50 data:0x7751c000-0x77526db0
/usr/lib/hpux32/libnsl.so.1
text:0xc03b2000-0xc04a5410 data:0x7752c000-0x77539da0
/usr/lib/hpux32/libxti.so.1
text:0xc04a6000-0xc04d4230 data:0x77529000-0x7752b420
/usr/lib/hpux32/libnss_dns.so.1
text:0xc0ceb000-0xc0cf8f00 data:0x7751b000-0x7751b370
VM Arguments:
jvm_args: -Xms256m -Xmx1024m -XX:PermSize=128m -XX:MaxPermSize=256m -Dbea.home=/d1/app/bea -Dweblogic.RootDirectory=/d1/app/bea/user_projects/domains/dev1 -Dweblogic.Name=server1 -Dweblogic.ListenAddress=myserver -Dweblogic.ListenPort=7003 -Dweblogic.management.server=t3://myserver:7001 -Dweblogic.system.BootIdentityFile=/d1/app/bea/user_projects/domains/dev1/servers/server1/security/boot.properties -Djavax.xml.soap.MessageFactory=weblogic.xml.saaj.MessageFactoryImpl -Dcom.sun.xml.namespace.QName.useCompatibleSerialVersionUID=1.0 -Djava.endorsed.dirs=/d1/home/wls/greencm/myapplication/core-4.0.0/blueEndorsed -Djava.awt.headless=true -Djava.security.auth.login.config=/d1/home/wls/greencm/myapplication/jaas.config -Djava.security.policy=/d1/app/bea/wlserver_10.3/server/lib/weblogic.policy
java_command: weblogic.Server
Launcher Type: SUN_STANDARD
Environment Variables:
JAVA_HOME=/opt/java1.6
CLASSPATH=/d1/app/bea/patch_wls1030/profiles/default/sys_manifest_classpath/weblogic_patch.jar:/d1/app/bea/patch_cie660/profiles/default/sys_manifest_classpath/weblogic_patch.jar:/opt/java1.6/lib/tools.jar:/d1/app/bea/wlserver_10.3/server/lib/weblogic_sp.jar:/d1/app/bea/wlserver_10.3/server/lib/weblogic.jar:/d1/app/bea/modules/features/weblogic.server.modules_10.3.0.0.jar:/d1/app/bea/wlserver_10.3/server/lib/webservices.jar:/d1/app/bea/modules/org.apache.ant_1.6.5/lib/ant-all.jar:/d1/app/bea/modules/net.sf.antcontrib_1.0.0.0_1-0b2/lib/ant-contrib.jar:
PATH=/d1/app/bea/wlserver_10.3/server/bin:/d1/app/bea/modules/org.apache.ant_1.6.5/bin:/opt/java1.6/jre/bin:/opt/java1.6/bin:/opt/java1.6/bin:/usr/bin:/usr/local/bin:/usr/bin:/usr/sbin:.
LD_LIBRARY_PATH=/opt/java1.6/jre/lib/IA64N:/opt/java1.6/jre/lib/IA64N/server:/opt/java1.6/jre/../lib/IA64N:/opt/java1.6/jre/lib/IA64N:/opt/java1.6/jre/lib/IA64N/server:/opt/java1.6/jre/../lib/IA64N
SHELL=/usr/bin/sh
SHLIB_PATH=:/d1/app/bea/wlserver_10.3/server/native/hpux11/IPF32:/d1/app/bea/wlserver_10.3/server/native/hpux11/IPF32/oci920_8
PTHREAD_COMPAT_MODE=0
Signal Handlers:
SIGSEGV: htp://libjvm.so+0xffed64a8, sa_mask[0]=0xbf7ffeff, sa_flags=0x00000051
SIGBUS : htp://libjvm.so+0xffed64a8, sa_mask[0]=0xbf7ffeff, sa_flags=0x00000051
SIGFPE : htp://libjvm.so+0xffed64a8, sa_mask[0]=0xbf7ffeff, sa_flags=0x00000050
SIGPIPE: htp://libjvm.so+0xffed64a8, sa_mask[0]=0xbf7ffeff, sa_flags=0x00000050
SIGXFSZ: htp://libjvm.so+0xffed64a8, sa_mask[0]=0xbf7ffeff, sa_flags=0x00000050
SIGILL : htp://libjvm.so+0xffed64a8, sa_mask[0]=0xbf7ffeff, sa_flags=0x00000051
SIGUSR1: SIG_DFL, sa_mask[0]=0x00000000, sa_flags=0x00000004
SIGUSR2: htp://libjvm.so+0xffed4dd8, sa_mask[0]=0x00000000, sa_flags=0x00000000
SIGHUP : SIG_IGN, sa_mask[0]=0x00000000, sa_flags=0x00000004
SIGINT : SIG_IGN, sa_mask[0]=0x00000000, sa_flags=0x00000004
SIGQUIT: htp://libjvm.so+0xffed1ee8, sa_mask[0]=0xbf7ffeff, sa_flags=0x00000040
SIGTERM: htp://libjvm.so+0xffed1ee8, sa_mask[0]=0xbf7ffeff, sa_flags=0x00000040
SIGRTMIN: htp://libjvm.so+0xffed4fa8, sa_mask[0]=0x00000000, sa_flags=0x00000010
SIGRTMIN+1: htp://libjvm.so+0xffed64a8, sa_mask[0]=0xbf7ffeff, sa_flags=0x00000050
S Y S T E M
OS:HP-UX
uname:HP-UX B.11.23 U ia64
rlimit: STACK 131072k, CORE 2097151k, NOFILE 4096, AS infinity
load average:0.45 0.14 0.06
CPU:total 2
Memory: 4k page, physical 4182800k(11892k free)
vm_info: Java HotSpot(TM) Server VM (11.0-b16-jre1.6.0.03-rc2) for hp-ux-ia64 JRE (1.6.0), built on Feb 6 2009 04:02:16 by "jinteg" with aCC
time: Tue Dec 1 01:33:12 2009
elapsed time: 9486 seconds
This is what is in /var/adm/syslog:
Nov 30 01:36:04 myserver syslog: Java out of memory messages are marked with pid:5031 in /var/adm/syslog/syslog.log.
Nov 30 01:36:04 myserver syslog: pid: 5031 4Kb 4Kb 4Kb PCT START/ 4Kb
DEV NO./
Nov 30 01:36:04 myserver syslog: pid: 5031 TYPE AVAIL USED FREE USED LIMIT RESERVE
PRI FILE NAME
Nov 30 01:36:04 myserver syslog: pid: 5031 dev 1048576 169912 878664 16.20% 0 -
1 64 (major) 2 (minor)
Nov 30 01:36:04 myserver syslog: pid: 5031 reserve - 579015 -579015
Nov 30 01:36:04 myserver syslog: pid: 5031 memory 1045700 411902 633798 39.39%
Sorry for any weird formatting and thanks.