JVM crash while memcpy during class load

Question

My JVM crashed the and the hs_err file showed that it crashed while attempting to load a class. Specifically while trying to memcpy ([libc.so.6+0x6aa2c] memcpy+0x1c). I looked at the .class file and was able to determine what class was being loaded.

But can any one tell me what could cause this or how i could determine more about the cause? If the JVM was out of memory wouldn't it throw an Error. Any insight is greatly appreciated.

I've included an excerpt from my hs_err file.

#
# An unexpected error has been detected by Java Runtime Environment:
#
#  SIGBUS (0x7) at pc=0x005aba2c, pid=20841, tid=2427227056
#
# Java VM: Java HotSpot(TM) Client VM (1.6.0_02-b05 mixed mode)
# Problematic frame:
# C  [libc.so.6+0x6aa2c]  memcpy+0x1c
#
# If you would like to submit a bug report, please visit:
#   http://java.sun.com/webapps/bugreport/crash.jsp
#

---------------  T H R E A D  ---------------

Current thread (0x90d0dc00):  JavaThread "ORDERHANDLER" [_thread_in_native, id=20881]

siginfo:si_signo=7, si_errno=0, si_code=2, si_addr=0x915e3000

Registers:
EAX=0x91218298, EBX=0xb7f2e71c, ECX=0x0000079b, EDX=0x915dfef2
ESP=0x90ac6a34, EBP=0x90ac6a60, ESI=0x915e2ffd, EDI=0x914f0a0d
EIP=0x005aba2c, CR2=0x915e3000, EFLAGS=0x00010206

Top of Stack: (sp=0x90ac6a34)
0x90ac6a34:   b7f29d4b 914ed930 915dff20 00004f49
0x90ac6a44:   082e7bc4 00006f6f 00004243 00004f49
0x90ac6a54:   b7f2e71c 080e3e54 00000000 90ac6a90
0x90ac6a64:   b7f29fbb 080e3b00 080e3e54 00000000
0x90ac6a74:   00000000 90d0dc00 00000000 d68dd1b6
0x90ac6a84:   b7f2e71c 90ac6ad8 90d0dcec 90ac6f00
0x90ac6a94:   b7f21169 080e3b00 90ac6ad8 0000002b
0x90ac6aa4:   0000002b 90ac6ad8 00000008 00000000

Instructions: (pc=0x005aba2c)
0x005aba1c:   8b 74 24 08 fc d1 e9 73 01 a4 d1 e9 73 02 66 a5
0x005aba2c:   f3 a5 89 c7 89 d6 8b 44 24 04 c3 90 90 90 90 90

Stack: [0x90a78000,0x90ac9000),  sp=0x90ac6a34,  free space=314k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
C  [libc.so.6+0x6aa2c]  memcpy+0x1c
C  [libzip.so+0xbfbb]  ZIP_GetEntry+0x10b
C  [libzip.so+0x3169]  Java_java_util_zip_ZipFile_getEntry+0xc9
J  java.util.zip.ZipFile.getEntry(JLjava/lang/String;Z)J
J  java.util.zip.ZipFile.getEntry(Ljava/lang/String;)Ljava/util/zip/ZipEntry;
J  java.util.jar.JarFile.getEntry(Ljava/lang/String;)Ljava/util/zip/ZipEntry;
J  sun.misc.URLClassPath$JarLoader.getResource(Ljava/lang/String;Z)Lsun/misc/Resource;
J  sun.misc.URLClassPath.getResource(Ljava/lang/String;Z)Lsun/misc/Resource;
J  java.net.URLClassLoader$1.run()Ljava/lang/Object;
v  ~StubRoutines::call_stub
V  [libjvm.so+0x20bbbd]
V  [libjvm.so+0x30a6b8]
V  [libjvm.so+0x20ba50]
V  [libjvm.so+0x26190b]
C  [libjava.so+0xaa5c]  Java_java_security_AccessController_doPrivileged__Ljava_security_PrivilegedExceptionAction_2Ljava_security_AccessControlContext_2+0x3
c
J  java.security.AccessController.doPrivileged(Ljava/security/PrivilegedExceptionAction;Ljava/security/AccessControlContext;)Ljava/lang/Object;
J  java.net.URLClassLoader.findClass(Ljava/lang/String;)Ljava/lang/Class;
J  java.lang.ClassLoader.loadClass(Ljava/lang/String;Z)Ljava/lang/Class;
J  sun.misc.Launcher$AppClassLoader.loadClass(Ljava/lang/String;Z)Ljava/lang/Class;
j  java.lang.ClassLoader.loadClass(Ljava/lang/String;)Ljava/lang/Class;+3
j  java.lang.ClassLoader.loadClassInternal(Ljava/lang/String;)Ljava/lang/Class;+2
v  ~StubRoutines::call_stub
V  [libjvm.so+0x20bbbd]
V  [libjvm.so+0x30a6b8]
V  [libjvm.so+0x20b6e1]
V  [libjvm.so+0x20b7ca]
V  [libjvm.so+0x367621]
V  [libjvm.so+0x3662a5]
V  [libjvm.so+0x365357]
V  [libjvm.so+0x365112]
V  [libjvm.so+0x1adb03]
V  [libjvm.so+0x1aeb32]
V  [libjvm.so+0x2d75cb]
V  [libjvm.so+0x2d8a94]
V  [libjvm.so+0x2d8a17]
V  [libjvm.so+0x1fe7f8]
j  com.aqua.foms.book.OrderHandler.handleNewOrder(Lcom/aqua/NmsApi/OrderMap;Lcom/aqua/api/AtsMessage;)V+221
j  com.aqua.foms.book.FMSNewOrderTask.execute()V+12
j  com.aqua.api.EEDefaultWorkerThread.run()V+96
v  ~StubRoutines::call_stub
V  [libjvm.so+0x20bbbd]
V  [libjvm.so+0x30a6b8]
V  [libjvm.so+0x20b4d0]
V  [libjvm.so+0x20b55d]
V  [libjvm.so+0x27b795]
V  [libjvm.so+0x383ef0]
V  [libjvm.so+0x30b5a9]
C  [libpthread.so.0+0x5371]

Java frames: (J=compiled Java code, j=interpreted, Vv=VM code)
J  java.util.zip.ZipFile.getEntry(JLjava/lang/String;Z)J
J  java.util.zip.ZipFile.getEntry(Ljava/lang/String;)Ljava/util/zip/ZipEntry;
J  java.util.jar.JarFile.getEntry(Ljava/lang/String;)Ljava/util/zip/ZipEntry;
J  sun.misc.URLClassPath$JarLoader.getResource(Ljava/lang/String;Z)Lsun/misc/Resource;
J  sun.misc.URLClassPath.getResource(Ljava/lang/String;Z)Lsun/misc/Resource;
J  java.net.URLClassLoader$1.run()Ljava/lang/Object;
v  ~StubRoutines::call_stub
J  java.security.AccessController.doPrivileged(Ljava/security/PrivilegedExceptionAction;Ljava/security/AccessControlContext;)Ljava/lang/Object;
J  java.net.URLClassLoader.findClass(Ljava/lang/String;)Ljava/lang/Class;
J  java.lang.ClassLoader.loadClass(Ljava/lang/String;Z)Ljava/lang/Class;
J  sun.misc.Launcher$AppClassLoader.loadClass(Ljava/lang/String;Z)Ljava/lang/Class;
j  java.lang.ClassLoader.loadClass(Ljava/lang/String;)Ljava/lang/Class;+3
j  java.lang.ClassLoader.loadClassInternal(Ljava/lang/String;)Ljava/lang/Class;+2
v  ~StubRoutines::call_stub
j  com.aqua.foms.book.OrderHandler.handleNewOrder(Lcom/aqua/NmsApi/OrderMap;Lcom/aqua/api/AtsMessage;)V+221
j  com.aqua.foms.book.FMSNewOrderTask.execute()V+12
j  com.aqua.api.EEDefaultWorkerThread.run()V+96
v  ~StubRoutines::call_stub

---------------  P R O C E S S  ---------------

Java Threads: ( => current thread )
  0x080c9c00 JavaThread "pool-1-thread-3" [_thread_blocked, id=18725]
  0x0824f800 JavaThread "pool-1-thread-2" [_thread_blocked, id=13693]
  0x91217c00 JavaThread "AquaSchedulerService_7" daemon [_thread_blocked, id=23675]
  0x91215c00 JavaThread "AquaSchedulerService_6" daemon [_thread_blocked, id=23001]
 0x91215400 JavaThread "AquaSchedulerService_5" daemon [_thread_blocked, id=22759]
  0x91213400 JavaThread "AquaSchedulerService_4" daemon [_thread_blocked, id=22410]
  0x91212c00 JavaThread "AquaSchedulerService_3" daemon [_thread_blocked, id=22262]
  0x08316400 JavaThread "pool-1-thread-1" [_thread_blocked, id=22260]
  0x0827d000 JavaThread "JmsConn_1_sender_0" daemon [_thread_blocked, id=21196]
  0x90d0cc00 JavaThread "Timer-0" [_thread_blocked, id=20882]
=>0x90d0dc00 JavaThread "ORDERHANDLER" [_thread_in_native, id=20881]
  0x90d0d400 JavaThread "TradeInviteMonitor" [_thread_blocked, id=20880]
  0x90d09c00 JavaThread "ROUTERT" [_thread_blocked, id=20878]
  0x90d09000 JavaThread "TIBCO EMS Session Dispatcher (33197)" [_thread_blocked, id=20877]
  0x08310800 JavaThread "DORDERHANDLER" [_thread_blocked, id=20874]
  0x90d01c00 JavaThread "Thread-12" daemon [_thread_blocked, id=20873]
  0x90d03000 JavaThread "Thread-11" daemon [_thread_in_native, id=20872]
  0x082e1c00 JavaThread "DELAYEDORDMON" [_thread_blocked, id=20871]
  0x082e8000 JavaThread "DBUPD" [_thread_blocked, id=20870]
  0x914e5000 JavaThread "pool-2-thread-1" [_thread_blocked, id=20869]
  0x914e3c00 JavaThread "StatusStatsEventDispatcherThread" [_thread_blocked, id=20868]
  0x082c8400 JavaThread "TimerQueue" daemon [_thread_blocked, id=20866]
  0x082ca000 JavaThread "MDATATHREAD" [_thread_blocked, id=20865]
  0x082c9400 JavaThread "AquaSchedulerService_2" daemon [_thread_blocked, id=20864]
  0x9122b000 JavaThread "DestroyJavaVM" [_thread_blocked, id=20843]
  0x91200800 JavaThread "FirmMatchingServer" [_thread_blocked, id=20863]
  0x914de800 JavaThread "TIBCO EMS TCPLink Reader (32084)" daemon [_thread_in_native, id=20861]
  0x9122a400 JavaThread "TIBCO EMS Connections Pinger" daemon [_thread_blocked, id=20859]
  0x914d4000 JavaThread "WDISTQ" [_thread_blocked, id=20858]
  0x9121f400 JavaThread "JmsConn_1_connector_0" daemon [_thread_blocked, id=20857]
  0x914d8000 JavaThread "JmsConn_1_receiver_0" daemon [_thread_blocked, id=20856]
  0x9149ac00 JavaThread "AquaSchedulerService_1" daemon [_thread_blocked, id=20855]
  0x9149b400 JavaThread "AquaSchedulerService_0" daemon [_thread_blocked, id=20854]
  0x9142a000 JavaThread "MySQL Statement Cancellation Timer" daemon [_thread_blocked, id=20852]
  0x91425c00 JavaThread "Dispatcher-Thread-0" daemon [_thread_blocked, id=20851]
  0x080bf800 JavaThread "Low Memory Detector" daemon [_thread_blocked, id=20849]
  0x080bdc00 JavaThread "CompilerThread0" daemon [_thread_blocked, id=20848]
  0x080bcc00 JavaThread "Signal Dispatcher" daemon [_thread_blocked, id=20847]
  0x080a9800 JavaThread "Finalizer" daemon [_thread_blocked, id=20846]
  0x080a8800 JavaThread "Reference Handler" daemon [_thread_blocked, id=20845]

Other Threads:
  0x080a5400 VMThread [id=20844]
  0x080c1000 WatcherThread [id=20850]

VM state:not at safepoint (normal execution)

VM Mutex/Monitor currently owned by a thread: None

Answer 1

Other than a plain ol. bug in the JVM(upgrade to the latest version and hope it doesn't happen again) - or some buggy 3. party libraries using JNI, there's 2 other "interesting" things that could cause this.

1. Hardware failure - bad RAM is often a good candidate ot a corrupted filesystem could cause of a flaky drive could be a culprit too.

2. If you're running on Solaris, you can get SIGBUS errors if somehow the class/jar file was truncated just when the JVM needs to access it in the cases the JVM mmaps the jar/class file.

Answer 2

we've seen similar errors. our current suspect is jar files which are re-written (by an upgrade process) while the process is running.