Tomcat JVM crash - EXCEPTION_ACCESS_VIOLATION (0xc0000005)

Question

I am getting the following jvm crash every few days running Tomcat 6.0.16 on java 1.6.0_07.

The effect is tomcat dies instantly.

Any ideas what this means?

An unexpected error has been detected by Java Runtime Environment:

  EXCEPTION_ACCESS_VIOLATION (0xc0000005) at pc=0x6d889bb3, pid=4228, tid=8264

 Java VM: Java HotSpot(TM) Client VM (10.0-b23 mixed mode windows-x86)
 Problematic frame:
V  [jvm.dll+0x19bb3]

 If you would like to submit a bug report, please visit:
  http://java.sun.com/webapps/bugreport/crash.jsp


---------------  T H R E A D  ---------------

Current thread (0x4ca02000):  VMThread [stack: 0x4ca90000,0x4cae0000] [id=8264]

siginfo: ExceptionCode=0xc0000005, reading address 0x616c632e

Registers:
EAX=0x4ce789a8, EBX=0x00000005, ECX=0x616c632e, EDX=0x00000000
ESP=0x4cadfc34, EBP=0x04598e70, ESI=0x00000000, EDI=0x4d068800
EIP=0x6d889bb3, EFLAGS=0x00010206

Top of Stack: (sp=0x4cadfc34)
0x4cadfc34:   4d66f034 4d66f034 4ca02428 4ca02420
0x4cadfc44:   00000005 ffffffff 4cadff74 77e61a60
0x4cadfc54:   77e61d48 6d88a32f 00000001 00000000
0x4cadfc64:   00000000 4ce78800 00292cd8 6da2f9df
0x4cadfc74:   4d66f034 00292cd8 4ca02420 00000000
0x4cadfc84:   004ef3fa 00000000 019db1de 4ca02814
0x4cadfc94:   6da2ec71 0000000b 002959d8 00000000
0x4cadfca4:   00000000 00000001 4f23e15b 4ca02428 

Instructions: (pc=0x6d889bb3)
0x6d889ba3:   d2 33 f6 85 c9 7e 67 8d 9b 00 00 00 00 8b 48 0c
0x6d889bb3:   8b 0c b1 39 29 75 09 8b 51 04 c7 02 00 00 00 00 


Stack: [0x4ca90000,0x4cae0000],  sp=0x4cadfc34,  free space=319k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
V  [jvm.dll+0x19bb3]

VM_Operation (0x4d66f034): RevokeBias, mode: safepoint, requested by thread 0x4ce78800


---------------  P R O C E S S  ---------------

Java Threads: ( => current thread )
  0x4cf59400 JavaThread "Thread-7" daemon [_thread_blocked, id=6240, stack(0x4d430000,0x4d480000)]
  0x4cef7c00 JavaThread "TP-Monitor" daemon [_thread_blocked, id=7080, stack(0x4d6c0000,0x4d710000)]
  0x4cd25800 JavaThread "TP-Processor4" daemon [_thread_in_native, id=7660, stack(0x4d670000,0x4d6c0000)]
  0x4ce78800 JavaThread "TP-Processor3" daemon [_thread_blocked, id=272, stack(0x4d620000,0x4d670000)]
  0x4d068800 JavaThread "TP-Processor2" daemon [_thread_blocked, id=620, stack(0x4d5d0000,0x4d620000)]
  0x4d068400 JavaThread "TP-Processor1" daemon [_thread_in_native, id=6132, stack(0x4d580000,0x4d5d0000)]
  0x4cf69400 JavaThread "ContainerBackgroundProcessor[StandardEngine[Catalina]]" daemon [_thread_blocked, id=6088, stack(0x4d530000,0x4d580000)]
  0x4d13a800 JavaThread "ObjectPoolCleaner" [_thread_blocked, id=5792, stack(0x4d4e0000,0x4d530000)]
  0x4d069000 JavaThread "ObjectPoolCleaner" [_thread_blocked, id=9296, stack(0x4d360000,0x4d3b0000)]
  0x4d091800 JavaThread "pool-1-thread-1" [_thread_blocked, id=7496, stack(0x4d310000,0x4d360000)]
  0x4ca1f800 JavaThread "Low Memory Detector" daemon [_thread_blocked, id=9860, stack(0x4cc70000,0x4ccc0000)]
  0x4ca1b000 JavaThread "CompilerThread0" daemon [_thread_blocked, id=5880, stack(0x4cc20000,0x4cc70000)]
  0x4ca19c00 JavaThread "Attach Listener" daemon [_thread_blocked, id=7064, stack(0x4cbd0000,0x4cc20000)]
  0x4ca18c00 JavaThread "Signal Dispatcher" daemon [_thread_blocked, id=756, stack(0x4cb80000,0x4cbd0000)]
  0x4ca09800 JavaThread "Finalizer" daemon [_thread_blocked, id=8240, stack(0x4cb30000,0x4cb80000)]
  0x4ca05400 JavaThread "Reference Handler" daemon [_thread_blocked, id=7888, stack(0x4cae0000,0x4cb30000)]
  0x00296400 JavaThread "main" [_thread_in_native, id=5184, stack(0x00840000,0x00890000)]

Other Threads:
=>0x4ca02000 VMThread [stack: 0x4ca90000,0x4cae0000] [id=8264]
  0x4ca2a400 WatcherThread [stack: 0x4ccc0000,0x4cd10000] [id=6520]

VM state:at safepoint (normal execution)

VM Mutex/Monitor currently owned by a thread:  ([mutex/lock_event])
[0x002959a8/0x000006e4] Threads_lock - owner thread: 0x4ca02000

Heap
 def new generation   total 72576K, used 33231K [0x02910000, 0x077d0000, 0x077d0000)
  eden space 64512K,  51% used [0x02910000, 0x04983fa0, 0x06810000)
  from space 8064K,   0% used [0x06ff0000, 0x06ff0000, 0x077d0000)
  to   space 8064K,   0% used [0x06810000, 0x06810000, 0x06ff0000)
 tenured generation   total 967936K, used 42171K [0x077d0000, 0x42910000, 0x42910000)
   the space 967936K,   4% used [0x077d0000, 0x0a0fee10, 0x0a0ff000, 0x42910000)
 compacting perm gen  total 19456K, used 19349K [0x42910000, 0x43c10000, 0x4a910000)
   the space 19456K,  99% used [0x42910000, 0x43bf5460, 0x43bf5600, 0x43c10000)
No shared spaces configured.

Dynamic libraries:
0x00400000 - 0x00423000     C:\platform\software\jdk1.6.0_07\bin\java.exe
0x7c800000 - 0x7c8c2000     C:\WINDOWS\system32\ntdll.dll
0x77e40000 - 0x77f42000     C:\WINDOWS\system32\kernel32.dll
0x7d1e0000 - 0x7d27c000     C:\WINDOWS\system32\ADVAPI32.dll
0x77c50000 - 0x77cef000     C:\WINDOWS\system32\RPCRT4.dll
0x76f50000 - 0x76f63000     C:\WINDOWS\system32\Secur32.dll
0x7c340000 - 0x7c396000     C:\platform\software\jdk1.6.0_07\jre\bin\msvcr71.dll
0x6d870000 - 0x6dac0000     C:\platform\software\jdk1.6.0_07\jre\bin\client\jvm.dll
0x77380000 - 0x77411000     C:\WINDOWS\system32\USER32.dll
0x77c00000 - 0x77c49000     C:\WINDOWS\system32\GDI32.dll
0x76aa0000 - 0x76acd000     C:\WINDOWS\system32\WINMM.dll
0x76290000 - 0x762ad000     C:\WINDOWS\system32\IMM32.DLL
0x6d320000 - 0x6d328000     C:\platform\software\jdk1.6.0_07\jre\bin\hpi.dll
0x76b70000 - 0x76b7b000     C:\WINDOWS\system32\PSAPI.DLL
0x6d820000 - 0x6d82c000     C:\platform\software\jdk1.6.0_07\jre\bin\verify.dll
0x6d3c0000 - 0x6d3df000     C:\platform\software\jdk1.6.0_07\jre\bin\java.dll
0x6d860000 - 0x6d86f000     C:\platform\software\jdk1.6.0_07\jre\bin\zip.dll
0x6d620000 - 0x6d633000     C:\platform\software\jdk1.6.0_07\jre\bin\net.dll
0x71c00000 - 0x71c17000     C:\WINDOWS\system32\WS2_32.dll
0x77ba0000 - 0x77bfa000     C:\WINDOWS\system32\msvcrt.dll
0x71bf0000 - 0x71bf8000     C:\WINDOWS\system32\WS2HELP.dll
0x71b20000 - 0x71b61000     C:\WINDOWS\system32\mswsock.dll
0x5f270000 - 0x5f2ca000     C:\WINDOWS\system32\hnetcfg.dll
0x71ae0000 - 0x71ae8000     C:\WINDOWS\System32\wshtcpip.dll
0x68000000 - 0x68035000     C:\WINDOWS\system32\rsaenh.dll
0x76920000 - 0x769e2000     C:\WINDOWS\system32\USERENV.dll
0x71c40000 - 0x71c97000     C:\WINDOWS\system32\netapi32.dll
0x76ed0000 - 0x76efa000     C:\WINDOWS\system32\DNSAPI.dll
0x76f70000 - 0x76f77000     C:\WINDOWS\System32\winrnr.dll
0x76f10000 - 0x76f3e000     C:\WINDOWS\system32\WLDAP32.dll
0x76f80000 - 0x76f85000     C:\WINDOWS\system32\rasadhlp.dll
0x6d800000 - 0x6d808000     C:\platform\software\jdk1.6.0_07\jre\bin\sunmscapi.dll
0x761b0000 - 0x76243000     C:\WINDOWS\system32\CRYPT32.dll
0x76190000 - 0x761a2000     C:\WINDOWS\system32\MSASN1.dll

VM Arguments:
jvm_args: -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Djava.util.logging.config.file=C:\platform\software\apache-tomcat-6.0.16\conf\logging.properties -Xms1024m -Xmx1024m -XX:MaxPermSize=128m -Djava.endorsed.dirs=C:\platform\software\apache-tomcat-6.0.16\endorsed -Dcatalina.base=C:\platform\software\apache-tomcat-6.0.16 -Dcatalina.home=C:\platform\software\apache-tomcat-6.0.16 -Djava.io.tmpdir=C:\platform\software\apache-tomcat-6.0.16\temp 
java_command: org.apache.catalina.startup.Bootstrap start
Launcher Type: SUN_STANDARD

Environment Variables:
JAVA_HOME=C:\platform\software\jdk1.6.0_07
JRE_HOME=C:\platform\software\jdk1.6.0_07  

OS=Windows_NT
PROCESSOR_IDENTIFIER=x86 Family 6 Model 26 Stepping 5, GenuineIntel



---------------  S Y S T E M  ---------------

OS: Windows Server 2003 family Build 3790 Service Pack 2

CPU:total 4 (1 cores per cpu, 1 threads per core) family 6 model 10 stepping 5, cmov, cx8, fxsr, mmx, sse, sse2, sse3, ssse3

Memory: 4k page, physical 2097151k(2097151k free), swap 4194303k(3879608k free)

vm_info: Java HotSpot(TM) Client VM (10.0-b23) for windows-x86 JRE (1.6.0_07-b06), built on Jun 10 2008 01:14:11 by "java_re" with MS VC++ 7.1

time: Mon Aug 09 08:34:32 2010
elapsed time: 34039 seconds

Answer 1

I would try upgrading the JVM to the latest version. There have been many fixes since then.

Answer 2

It basically crashing with segfault in the JVM code. ExceptionCode=0xc0000005 means segfault. JVM thread(tid=0x4ca02000) tried to write at the address pointed by ECX=0x616c632e register,which looks to be inaccessible,and caused the segfault. I can see two reason based on the info provided,either its a object corruption issue due to access garbage collection or compaction in a java heap(possibly perm region--just a guess based on the occupancy shown in the heap stats) or may be a object corruption due to JIT compiled code.First is kinda difficult to figure out because it needs the system dump analysis,but looks like second one is easy to check by disabling JIT using -Xint option.You can try this and if it resolves the issue than possibly JIT is doing something bad.You can check in the latest release if something is fixed in JIT in recent time and can upgrade to latest version.