Segfaulting Java process

Question

I have a Java process that is working on some large data set in memory. I've seen it crash with a SIGSEGV signal sometimes, so i was wondering some potential causes and fixes could do.

Caues:

• JVM bug
• Native library bug (e.g pthreads etc)
• JNI bug in user code

Fixes:

• Upgrade to new JVM

In my particular case, this is the output form the log file (pruned)

# A fatal error has been detected by the Java Runtime Environment:
#
#  SIGSEGV (0xb) at pc=0x00002aaaaacd1b94, pid=32116, tid=1086544208
#
# JRE version: 6.0_14-b08
# Java VM: Java HotSpot(TM) 64-Bit Server VM (14.0-b16 mixed mode linux-amd64 )
# Problematic frame:
# C  [libpthread.so.0+0xab94]  pthread_cond_timedwait+0x154
#
# If you would like to submit a bug report, please visit:
#   http://java.sun.com/webapps/bugreport/crash.jsp
#

---------------  T H R E A D  ---------------

Current thread (0x00002aacaad41000):  WatcherThread [stack: 0x0000000040b35000,0x0000000040c36000] [id=32141]

siginfo:si_signo=SIGSEGV: si_errno=0, si_code=1 (SEGV_MAPERR), si_addr=0x00002aabc40008c0

Registers:
RAX=0x0000000000000000, RBX=0x0000000000000000, RCX=0x0000000000000000, RDX=0x0000000000000002
RSP=0x0000000040c34cc0, RBP=0x0000000040c34d80, RSI=0x0000000000000001, RDI=0x00002aabc40008c0
R8 =0x00002aacaad42528, R9 =0x0000000000000000, R10=0x0000000040c34cd8, R11=0x0000000000000202
R12=0x0000000000000001, R13=0x0000000040c34d40, R14=0xffffffffffffff92, R15=0x00002aacaad42550
RIP=0x00002aaaaacd1b94, EFL=0x0000000000010246, CSGSFS=0x000000000000e033, ERR=0x0000000000000006
  TRAPNO=0x000000000000000e

Top of Stack: (sp=0x0000000040c34cc0)
0x0000000040c34cc0:   0000000000000000 00002aabc40008c0
0x0000000040c34cd0:   00002aacaad42528 0000000000000000
0x0000000040c34ce0:   0000000002fae0e0 0000000000000000
0x0000000040c34cf0:   00002aaaaacd1750 0000000040c34cc0
0x0000000040c34d00:   00002aacaad42528 0000000000000000
0x0000000040c34d10:   00002aacaad42528 00002aacaad42500
0x0000000040c34d20:   0000000000000032 00002aaaabadf876
0x0000000040c34d30:   fffffffdaad40e80 0000000040c34d40
0x0000000040c34d40:   000000004bbb7166 0000000015f07098
0x0000000040c34d50:   0000000040c34d80 00138cd32df59cce
0x0000000040c34d60:   431bde82d7b634db 00002aacaad429c0
0x0000000040c34d70:   0000000000000032 00002aacaad429c0
0x0000000040c34d80:   0000000040c34e00 00002aaaabadda6d
0x0000000040c34d90:   0000000040c34da0 00002aacaad42500
0x0000000040c34da0:   00002aacaad429c0 00002aaa00000002
0x0000000040c34db0:   0000000000000001 0000000000000002
0x0000000040c34dc0:   0000000040c34dd0 00002aaaabb6f613
0x0000000040c34dd0:   0000000040c34e00 00002aacaad41000
0x0000000040c34de0:   0000000000000032 00002aacaad429c0
0x0000000040c34df0:   00002aacaad41000 0000000000001000
0x0000000040c34e00:   0000000040c34e60 00002aaaabbc39fb
0x0000000040c34e10:   0000000040c34e40 00002aaaabab868f
0x0000000040c34e20:   00002aacaad41000 00002aacaad42aa0
0x0000000040c34e30:   00002aacaad42aa0 00002aaaabe10630
0x0000000040c34e40:   00002aaaabe10630 00002aacaad42aa0
0x0000000040c34e50:   00002aacaad429c0 00002aacaad41000
0x0000000040c34e60:   0000000040c35130 00002aaaabadff9f
0x0000000040c34e70:   0000000000000000 0000000000000000
0x0000000040c34e80:   0000000000000000 0000000000000000
0x0000000040c34e90:   0000000000000000 0000000000000000
0x0000000040c34ea0:   0000000000000000 0000000000000000
0x0000000040c34eb0:   0000000000000000 0000000000000000 

Instructions: (pc=0x00002aaaaacd1b94)
0x00002aaaaacd1b84:   88 22 00 00 48 8b 7c 24 08 be 01 00 00 00 31 c0
0x00002aaaaacd1b94:   f0 0f b1 37 0f 85 e8 00 00 00 8b 57 2c 48 8b 47 

Stack: [0x0000000040b35000,0x0000000040c36000],  sp=0x0000000040c34cc0,  free space=1023k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
C  [libpthread.so.0+0xab94]  pthread_cond_timedwait+0x154
V  [libjvm.so+0x594a6d]
V  [libjvm.so+0x67a9fb]
V  [libjvm.so+0x596f9f]


---------------  P R O C E S S  ---------------

Java Threads: ( => current thread )
  0x00002aacaad3f000 JavaThread "Low Memory Detector" daemon [_thread_blocked, id=32140, stack(0x0000000040a34000,0x0000000040b35000)]
  0x00002aacaad3c000 JavaThread "CompilerThread1" daemon [_thread_blocked, id=32139, stack(0x0000000040933000,0x0000000040a34000)]
  0x00002aacaad37800 JavaThread "CompilerThread0" daemon [_thread_blocked, id=32138, stack(0x0000000040832000,0x0000000040933000)]
  0x00002aacaad36800 JavaThread "Signal Dispatcher" daemon [_thread_blocked, id=32137, stack(0x0000000040731000,0x0000000040832000)]
  0x00002aacaab7d800 JavaThread "Finalizer" daemon [_thread_blocked, id=32136, stack(0x0000000040630000,0x0000000040731000)]
  0x00002aacaab7b800 JavaThread "Reference Handler" daemon [_thread_blocked, id=32135, stack(0x000000004052f000,0x0000000040630000)]
  0x0000000040115800 JavaThread "main" [_thread_blocked, id=32117, stack(0x000000004012b000,0x000000004022c000)]

Other Threads:
  0x00002aacaab75000 VMThread [stack: 0x000000004042e000,0x000000004052f000] [id=32134]
=>0x00002aacaad41000 WatcherThread [stack: 0x0000000040b35000,0x0000000040c36000] [id=32141]

VM state:at safepoint (normal execution)

VM Mutex/Monitor currently owned by a thread:  ([mutex/lock_event])
[0x0000000040112e80] Threads_lock - owner thread: 0x00002aacaab75000
[0x0000000040113380] Heap_lock - owner thread: 0x0000000040115800

Heap
 PSYoungGen      total 1854528K, used 1029248K [0x00002aac025a0000, 0x00002aaca8340000, 0x00002aaca9040000)
  eden space 1029248K, 100% used [0x00002aac025a0000,0x00002aac412c0000,0x00002aac412c0000)
  from space 825280K, 0% used [0x00002aac412c0000,0x00002aac412c0000,0x00002aac738b0000)
  to   space 812800K, 0% used [0x00002aac76980000,0x00002aac76980000,0x00002aaca8340000)
 PSOldGen        total 4423680K, used 4423651K [0x00002aaab5040000, 0x00002aabc3040000, 0x00002aac025a0000)
  object space 4423680K, 99% used [0x00002aaab5040000,0x00002aabc3038fe8,0x00002aabc3040000)
 PSPermGen       total 21248K, used 5848K [0x00002aaaafc40000, 0x00002aaab1100000, 0x00002aaab5040000)
  object space 21248K, 27% used [0x00002aaaafc40000,0x00002aaab01f61f0,0x00002aaab1100000)

Dynamic libraries:
40000000-40009000 r-xp 00000000 08:01 313415                             /usr/java/jdk1.6.0_14/bin/java
40108000-4010a000 rwxp 00008000 08:01 313415                             /usr/java/jdk1.6.0_14/bin/java
4010a000-4012b000 rwxp 4010a000 00:00 0                                  [heap]
4012b000-4012e000 ---p 4012b000 00:00 0 
4012e000-4022c000 rwxp 4012e000 00:00 0 
4022c000-4022d000 ---p 4022c000 00:00 0 
4022d000-4032d000 rwxp 4022d000 00:00 0 
4032d000-4032e000 ---p 4032d000 00:00 0 
4032e000-4042e000 rwxp 4032e000 00:00 0 
4042e000-4042f000 ---p 4042e000 00:00 0 
4042f000-4052f000 rwxp 4042f000 00:00 0 
4052f000-40532000 ---p 4052f000 00:00 0 
40532000-40630000 rwxp 40532000 00:00 0 
40630000-40633000 ---p 40630000 00:00 0 
40633000-40731000 rwxp 40633000 00:00 0 
40731000-40734000 ---p 40731000 00:00 0 
40734000-40832000 rwxp 40734000 00:00 0 
40832000-40835000 ---p 40832000 00:00 0 
40835000-40933000 rwxp 40835000 00:00 0 
40933000-40936000 ---p 40933000 00:00 0 
40936000-40a34000 rwxp 40936000 00:00 0 
40a34000-40a37000 ---p 40a34000 00:00 0 
40a37000-40b35000 rwxp 40a37000 00:00 0 
40b35000-40b36000 ---p 40b35000 00:00 0 
40b36000-40c36000 rwxp 40b36000 00:00 0 
2aaaaaaab000-2aaaaaac6000 r-xp 00000000 08:01 49198                      /lib64/ld-2.7.so
2aaaaaac6000-2aaaaaac7000 rwxp 2aaaaaac6000 00:00 0 
2aaaaaac7000-2aaaaaad0000 r-xs 0006d000 08:10 29851669                   /mnt/home/jatten/workspace/common/build/lib/common.jar
2aaaaaad2000-2aaaaaad3000 rwxp 2aaaaaad2000 00:00 0 
2aaaaaad3000-2aaaaaae0000 r-xp 00000000 08:01 315357                     /usr/java/jdk1.6.0_14/jre/lib/amd64/libverify.so
2aaaaaae0000-2aaaaabdf000 ---p 0000d000 08:01 315357                     /usr/java/jdk1.6.0_14/jre/lib/amd64/libverify.so
2aaaaabdf000-2aaaaabe2000 rwxp 0000c000 08:01 315357                     /usr/java/jdk1.6.0_14/jre/lib/amd64/libverify.so
2aaaaabe2000-2aaaaac0a000 rwxp 2aaaaabe2000 00:00 0 
2aaaaac0a000-2aaaaac0f000 r-xs 0003a000 08:10 30326840                   /mnt/home/jatten/workspace/common_ml20010405/build/lib/common_ml.jar
2aaaaac0f000-2aaaaac12000 r-xs 00020000 08:10 29786222                   /mnt/home/jatten/pagescorer.jar
2aaaaacc5000-2aaaaacc6000 r-xp 0001a000 08:01 49198                      /lib64/ld-2.7.so
2aaaaacc6000-2aaaaacc7000 rwxp 0001b000 08:01 49198                      /lib64/ld-2.7.so
2aaaaacc7000-2aaaaacdd000 r-xp 00000000 08:01 49280                      /lib64/libpthread-2.7.so
2aaaaacdd000-2aaaaaedc000 ---p 00016000 08:01 49280                      /lib64/libpthread-2.7.so
2aaaaaedc000-2aaaaaedd000 r-xp 00015000 08:01 49280                      /lib64/libpthread-2.7.so
2aaaaaedd000-2aaaaaede000 rwxp 00016000 08:01 49280                      /lib64/libpthread-2.7.so
2aaaaaede000-2aaaaaee2000 rwxp 2aaaaaede000 00:00 0 
2aaaaaee2000-2aaaaaee9000 r-xp 00000000 08:01 315360                     /usr/java/jdk1.6.0_14/jre/lib/amd64/jli/libjli.so
2aaaaaee9000-2aaaaafea000 ---p 00007000 08:01 315360                     /usr/java/jdk1.6.0_14/jre/lib/amd64/jli/libjli.so
2aaaaafea000-2aaaaafec000 rwxp 00008000 08:01 315360                     /usr/java/jdk1.6.0_14/jre/lib/amd64/jli/libjli.so
2aaaaafec000-2aaaaafee000 r-xp 00000000 08:01 49240                      /lib64/libdl-2.7.so
2aaaaafee000-2aaaab1ee000 ---p 00002000 08:01 49240                      /lib64/libdl-2.7.so
2aaaab1ee000-2aaaab1ef000 r-xp 00002000 08:01 49240                      /lib64/libdl-2.7.so
2aaaab1ef000-2aaaab1f0000 rwxp 00003000 08:01 49240                      /lib64/libdl-2.7.so
2aaaab1f0000-2aaaab1f1000 rwxp 2aaaab1f0000 00:00 0 
2aaaab1f1000-2aaaab33e000 r-xp 00000000 08:01 49219                      /lib64/libc-2.7.so
2aaaab33e000-2aaaab53e000 ---p 0014d000 08:01 49219                      /lib64/libc-2.7.so
2aaaab53e000-2aaaab542000 r-xp 0014d000 08:01 49219                      /lib64/libc-2.7.so
2aaaab542000-2aaaab543000 rwxp 00151000 08:01 49219                      /lib64/libc-2.7.so
2aaaab543000-2aaaab549000 rwxp 2aaaab543000 00:00 0 
2aaaab549000-2aaaabca7000 r-xp 00000000 08:01 315371                     /usr/java/jdk1.6.0_14/jre/lib/amd64/server/libjvm.so
2aaaabca7000-2aaaabda6000 ---p 0075e000 08:01 315371                     /usr/java/jdk1.6.0_14/jre/lib/amd64/server/libjvm.so
2aaaabda6000-2aaaabf1e000 rwxp 0075d000 08:01 315371                     /usr/java/jdk1.6.0_14/jre/lib/amd64/server/libjvm.so
2aaaabf1e000-2aaaabf5c000 rwxp 2aaaabf1e000 00:00 0 
2aaaabf67000-2aaaabfe9000 r-xp 00000000 08:01 49263                      /lib64/libm-2.7.so
2aaaabfe9000-2aaaac1e8000 ---p 00082000 08:01 49263                      /lib64/libm-2.7.so
2aaaac1e8000-2aaaac1e9000 r-xp 00081000 08:01 49263                      /lib64/libm-2.7.so
2aaaac1e9000-2aaaac1ea000 rwxp 00082000 08:01 49263                      /lib64/libm-2.7.so
2aaaac1ea000-2aaaac1f2000 r-xp 00000000 08:01 49283                      /lib64/librt-2.7.so
2aaaac1f2000-2aaaac3f1000 ---p 00008000 08:01 49283                      /lib64/librt-2.7.so
2aaaac3f1000-2aaaac3f2000 r-xp 00007000 08:01 49283                      /lib64/librt-2.7.so
2aaaac3f2000-2aaaac3f3000 rwxp 00008000 08:01 49283                      /lib64/librt-2.7.so
2aaaac3f3000-2aaaac41c000 r-xp 00000000 08:01 315336                     /usr/java/jdk1.6.0_14/jre/lib/amd64/libjava.so
2aaaac41c000-2aaaac51b000 ---p 00029000 08:01 315336                     /usr/java/jdk1.6.0_14/jre/lib/amd64/libjava.so
2aaaac51b000-2aaaac522000 rwxp 00028000 08:01 315336                     /usr/java/jdk1.6.0_14/jre/lib/amd64/libjava.so
2aaaac522000-2aaaac523000 ---p 2aaaac522000 00:00 0 
2aaaac523000-2aaaac524000 rwxp 2aaaac523000 00:00 0 
2aaaac52d000-2aaaac542000 r-xp 00000000 08:01 49265                      /lib64/libnsl-2.7.so
2aaaac542000-2aaaac741000 ---p 00015000 08:01 49265                      /lib64/libnsl-2.7.so
2aaaac741000-2aaaac742000 r-xp 00014000 08:01 49265                      /lib64/libnsl-2.7.so
2aaaac742000-2aaaac743000 rwxp 00015000 08:01 49265                      /lib64/libnsl-2.7.so
2aaaac743000-2aaaac745000 rwxp 2aaaac743000 00:00 0 
2aaaac745000-2aaaac74c000 r-xp 00000000 08:01 315362                     /usr/java/jdk1.6.0_14/jre/lib/amd64/native_threads/libhpi.so
2aaaac74c000-2aaaac84d000 ---p 00007000 08:01 315362                     /usr/java/jdk1.6.0_14/jre/lib/amd64/native_threads/libhpi.so
2aaaac84d000-2aaaac84f000 rwxp 00008000 08:01 315362                     /usr/java/jdk1.6.0_14/jre/lib/amd64/native_threads/libhpi.so
2aaaac84f000-2aaaac850000 rwxp 2aaaac84f000 00:00 0 
2aaaac850000-2aaaac858000 rwxs 00000000 08:01 229379                     /tmp/hsperfdata_jatten/32116
2aaaac85b000-2aaaac865000 r-xp 00000000 08:01 49269                      /lib64/libnss_files-2.7.so
2aaaac865000-2aaaaca64000 ---p 0000a000 08:01 49269                      /lib64/libnss_files-2.7.so
2aaaaca64000-2aaaaca65000 r-xp 00009000 08:01 49269                      /lib64/libnss_files-2.7.so
2aaaaca65000-2aaaaca66000 rwxp 0000a000 08:01 49269                      /lib64/libnss_files-2.7.so
2aaaaca66000-2aaaaca74000 r-xp 00000000 08:01 315358                     /usr/java/jdk1.6.0_14/jre/lib/amd64/libzip.so
2aaaaca74000-2aaaacb76000 ---p 0000e000 08:01 315358                     /usr/java/jdk1.6.0_14/jre/lib/amd64/libzip.so
2aaaacb76000-2aaaacb79000 rwxp 00010000 08:01 315358                     /usr/java/jdk1.6.0_14/jre/lib/amd64/libzip.so
2aaaacb79000-2aaaacdea000 rwxp 2aaaacb79000 00:00 0 
2aaaacdea000-2aaaafb7a000 rwxp 2aaaacdea000 00:00 0 
2aaaafb7a000-2aaaafb84000 rwxp 2aaaafb7a000 00:00 0 
2aaaafb84000-2aaaafc3a000 rwxp 2aaaafb84000 00:00 0 
2aaaafc40000-2aaab1100000 rwxp 2aaaafc40000 00:00 0 
2aaab1100000-2aaab5040000 rwxp 2aaab1100000 00:00 0 
2aaab5040000-2aabc3040000 rwxp 2aaab5040000 00:00 0 
2aac025a0000-2aaca8340000 rwxp 2aac025a0000 00:00 0 
2aaca8340000-2aaca9040000 rwxp 2aaca8340000 00:00 0 
2aaca9040000-2aaca904b000 rwxp 2aaca9040000 00:00 0 
2aaca904b000-2aaca906a000 rwxp 2aaca904b000 00:00 0 
2aaca906a000-2aaca98da000 rwxp 2aaca906a000 00:00 0 
2aaca98da000-2aaca9ad4000 rwxp 2aaca98da000 00:00 0 
2aaca9ad4000-2aacaa004000 rwxp 2aaca9ad4000 00:00 0 
2aacaa004000-2aacaa00a000 rwxp 2aacaa004000 00:00 0 
2aacaa00a000-2aacaa87b000 rwxp 2aacaa00a000 00:00 0 
2aacaa87b000-2aacaaa76000 rwxp 2aacaa87b000 00:00 0 
2aacaaa76000-2aacaaa81000 rwxp 2aacaaa76000 00:00 0 
2aacaaa81000-2aacaaaa0000 rwxp 2aacaaa81000 00:00 0 
2aacaaaa0000-2aacaaba0000 rwxp 2aacaaaa0000 00:00 0 
2aacaaba0000-2aacaad36000 r-xs 02fb1000 08:01 315318                     /usr/java/jdk1.6.0_14/jre/lib/rt.jar
2aacaad36000-2aacaaf36000 rwxp 2aacaad36000 00:00 0 
2aacaaf36000-2aacaaf49000 r-xp 00000000 08:01 315349                     /usr/java/jdk1.6.0_14/jre/lib/amd64/libnet.so
2aacaaf49000-2aacab04a000 ---p 00013000 08:01 315349                     /usr/java/jdk1.6.0_14/jre/lib/amd64/libnet.so
2aacab04a000-2aacab04d000 rwxp 00014000 08:01 315349                     /usr/java/jdk1.6.0_14/jre/lib/amd64/libnet.so
2aacab058000-2aacab05c000 r-xp 00000000 08:01 49268                      /lib64/libnss_dns-2.7.so
2aacab05c000-2aacab25b000 ---p 00004000 08:01 49268                      /lib64/libnss_dns-2.7.so
2aacab25b000-2aacab25c000 r-xp 00003000 08:01 49268                      /lib64/libnss_dns-2.7.so
2aacab25c000-2aacab25d000 rwxp 00004000 08:01 49268                      /lib64/libnss_dns-2.7.so
2aacab25d000-2aacab26e000 r-xp 00000000 08:01 49282                      /lib64/libresolv-2.7.so
2aacab26e000-2aacab46e000 ---p 00011000 08:01 49282                      /lib64/libresolv-2.7.so
2aacab46e000-2aacab46f000 r-xp 00011000 08:01 49282                      /lib64/libresolv-2.7.so
2aacab46f000-2aacab470000 rwxp 00012000 08:01 49282                      /lib64/libresolv-2.7.so
2aacab470000-2aacab572000 rwxp 2aacab470000 00:00 0 
2aacab572000-2aacab57e000 r-xs 00081000 08:10 29851828                   /mnt/home/jatten/workspace/common/lib/google-collect-1.0.jar
2aacab57e000-2aacab585000 r-xs 000aa000 08:10 29851946                   /mnt/home/jatten/workspace/common/lib/mysql-connector-java-5.1.8-bin.jar
2aacab585000-2aacab58d000 r-xs 00028000 08:10 29851949                   /mnt/home/jatten/workspace/common/lib/xml-apis.jar
2aacab58d000-2aacab591000 r-xs 0002f000 08:10 29851947                   /mnt/home/jatten/workspace/common/lib/commons-beanutils-core-1.8.2.jar
2aacab591000-2aacab59e000 r-xs 0007f000 08:10 29851943                   /mnt/home/jatten/workspace/common/lib/commons-collections-3.2.jar
2aacab59e000-2aacab5a3000 r-xs 00026000 08:10 29851942                   /mnt/home/jatten/workspace/common/lib/httpcore-4.0.jar
2aacab5a3000-2aacab5a9000 r-xs 00030000 08:10 29851932                   /mnt/home/jatten/workspace/common/lib/junit-dep-4.8.1.jar
2aacab5a9000-2aacab5ac000 r-xs 00011000 08:10 29851922                   /mnt/home/jatten/workspace/common/lib/servlet.jar
2aacab5ac000-2aacab5ae000 r-xs 00009000 08:10 29851937                   /mnt/home/jatten/workspace/common/lib/gsb.jar
2aacab5ae000-2aacab5b5000 r-xs 00059000 08:10 29851930                   /mnt/home/jatten/workspace/common/lib/log4j-1.2.15.jar
2aacab5b5000-2aacab6b5000 rwxp 2aacab5b5000 00:00 0 
2aacab6b5000-2aacab6b7000 r-xs 00009000 08:10 29851956                   /mnt/home/jatten/workspace/common/lib/gsb-src.jar
2aacab6b7000-2aacab7b7000 rwxp 2aacab6b7000 00:00 0 
2aacab7b7000-2aacab7cf000 r-xs 00115000 08:10 29851938                   /mnt/home/jatten/workspace/common/lib/xercesImpl.jar
2aacab7cf000-2aacab7d1000 r-xs 00009000 08:10 29851957                   /mnt/home/jatten/workspace/common/lib/velocity-tools-view-1.0.jar
2aacab7d1000-2aacab7d3000 r-xs 00009000 08:10 29851939                   /mnt/home/jatten/workspace/common/lib/commons-cli-1.2.jar
2aacab7d3000-2aacab7d9000 r-xs 00034000 08:10 29851955                   /mnt/home/jatten/workspace/common/lib/junit-4.8.1.jar
2aacab7d9000-2aacab7db000 r-xs 0000e000 08:10 29851917                   /mnt/home/jatten/workspace/common/lib/jakarta-oro-2.0.8.jar
2aacab7db000-2aacab858000 r-xs 0031d000 08:10 29851916                   /mnt/home/jatten/workspace/common/lib/poi-ooxml-schemas-3.6-20091214.jar
2aacab858000-2aacab85c000 r-xs 00028000 08:10 29851936                   /mnt/home/jatten/workspace/common/lib/httpcore-nio-4.0.jar
2aacab85c000-2aacab85e000 r-xs 00005000 08:10 29851940                   /mnt/home/jatten/workspace/common/lib/commons-beanutils-bean-collections-1.8.2.jar
2aacab85e000-2aacab864000 r-xs 00059000 08:10 29851919                   /mnt/home/jatten/workspace/common/lib/mail-1.4.jar
2aacab864000-2aacab866000 r-xs 0000d000 08:10 29851950                   /mnt/home/jatten/workspace/common/lib/commons-logging-1.1.1.jar
2aacab866000-2aacab86c000 r-xs 00045000 08:10 29851924                   /mnt/home/jatten/workspace/common/lib/commons-httpclient-3.1.jar
2aacab86c000-2aacab877000 r-xs 00074000 08:10 29851931                   /mnt/home/jatten/workspace/common/lib/velocity-dep-1.4.jar
2aacab877000-2aacab87f000 r-xs 00051000 08:10 29851954                   /mnt/home/jatten/workspace/common/lib/velocity-1.4.jar
2aacab87f000-2aacab884000 r-xs 00034000 08:10 29851958                   /mnt/home/jatten/workspace/common/lib/commons-beanutils-1.8.2.jar
2aacab884000-2aacab889000 r-xs 00048000 08:10 29851918                   /mnt/home/jatten/workspace/common/lib/dom4j-1.6.1.jar
2aacab889000-2aacab8c6000 r-xs 0024f000 08:10 29851914                   /mnt/home/jatten/workspace/common/lib/xmlbeans-2.3.0.jar
2aacab8c6000-2aacab8cb000 r-xs 00033000 08:10 29851929                   /mnt/home/jatten/workspace/common/lib/xmemcached-1.2.3.jar
2aacab8cb000-2aacab8cd000 r-xs 00005000 08:10 29851928                   /mnt/home/jatten/workspace/common/lib/org.hamcrest.core_1.1.0.v20090501071000.jar
2aacab8cd000-2aacab8d0000 r-xs 0000a000 08:10 29851944                   /mnt/home/jatten/workspace/common/lib/persistence-api-1.0.jar
2aacab8d0000-2aacab8d6000 r-xs 0005f000 08:10 29851926                   /mnt/home/jatten/workspace/common/lib/poi-ooxml-3.6-20091214.jar
2aacab8d6000-2aacab8d7000 r-xs 0002b000 08:10 29851951                   /mnt/home/jatten/workspace/common/lib/maxmind.jar
2aacab8d7000-2aacab8d8000 r-xs 00002000 08:10 29851935                   /mnt/home/jatten/workspace/common/lib/jackson-jaxrs-1.2.0.jar
2aacab8d8000-2aacab8d9000 r-xs 00002000 08:10 29851913                   /mnt/home/jatten/workspace/common/lib/slf4j-log4j12-1.5.6.jar
2aacab8d9000-2aacab8dd000 r-xs 00025000 08:10 29851945                   /mnt/home/jatten/workspace/common/lib/yanf4j-1.1.1.jar
2aacab8dd000-2aacab8df000 r-xs 00003000 08:10 29851952                   /mnt/home/jatten/workspace/common/lib/clickstream-1.0.2.jar
2aacab8df000-2aacab8e1000 r-xs 00004000 08:10 29851953                   /mnt/home/jatten/workspace/common/lib/slf4j-api-1.5.6.jar
2aacab8e1000-2aacab8e9000 r-xs 0004d000 08:10 29851920                   /mnt/home/jatten/workspace/common/lib/jackson-mapper-asl-1.2.0.jar
2aacab8e9000-2aacab8ed000 r-xs 0001f000 08:10 29851925                   /mnt/home/jatten/workspace/common/lib/jackson-core-asl-1.2.0.jar
2aacab8ed000-2aacab8f1000 r-xs 0001b000 08:10 29851912                   /mnt/home/jatten/workspace/common/lib/oscache-2.3.jar
2aacab8f1000-2aacab90c000 r-xs 0015d000 08:10 29851927                   /mnt/home/jatten/workspace/common/lib/poi-3.6-20091214.jar
2aacab90c000-2aacab911000 r-xs 00040000 08:10 29851831                   /mnt/home/jatten/workspace/common/lib/commons-lang-2.5.jar
2aacab911000-2aacab914000 r-xs 00012000 08:10 29851923                   /mnt/home/jatten/workspace/common/lib/jgooglesafebrowser-0.1a.2.jar
2aacab914000-2aacab918000 r-xs 00023000 08:10 29851933                   /mnt/home/jatten/workspace/common/lib/gson-1.3.jar
2aacab918000-2aacabb18000 rwxp 2aacab918000 00:00 0 
2aacabb82000-2aacabd82000 rwxp 2aacabb82000 00:00 0 
2aacabe05000-2aacaf204000 rwxp 2aacabe05000 00:00 0 
7fffaa12a000-7fffaa141000 rwxp 7fffaa12a000 00:00 0                      [stack]
ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0                  [vdso]

VM Arguments:
jvm_args: -Xmx8000M 
java_command: com.scorers.ModelImplementingPageScorer -t data/data/golds/adult.all.json -b 18 -s data/models/pagetext.binary.
adult.april6.all.model -m com.models.MultiClassUpdateableModel -p 30 --goldsilver -v --cat adult --fakeinput -e /mnt/tmp/xyz.15647.pageo
bjects.txt -o
Launcher Type: SUN_STANDARD

Environment Variables:
JAVA_HOME=/usr/java/jdk1.6.0_14
PATH=/usr/kerberos/bin:/usr/local/bin:/bin:/usr/bin:/home/jatten/bin
LD_LIBRARY_PATH=/usr/java/jdk1.6.0_14/jre/lib/amd64/server:/usr/java/jdk1.6.0_14/jre/lib/amd64:/usr/java/jdk1.6.0_14/jre/../lib/amd64
SHELL=/bin/bash

Signal Handlers:
SIGSEGV: [libjvm.so+0x6bd980], sa_mask[0]=0x7ffbfeff, sa_flags=0x10000004
SIGBUS: [libjvm.so+0x6bd980], sa_mask[0]=0x7ffbfeff, sa_flags=0x10000004
SIGFPE: [libjvm.so+0x594cc0], sa_mask[0]=0x7ffbfeff, sa_flags=0x10000004
SIGPIPE: [libjvm.so+0x594cc0], sa_mask[0]=0x7ffbfeff, sa_flags=0x10000004
SIGXFSZ: [libjvm.so+0x594cc0], sa_mask[0]=0x7ffbfeff, sa_flags=0x10000004
SIGILL: [libjvm.so+0x594cc0], sa_mask[0]=0x7ffbfeff, sa_flags=0x10000004
SIGUSR1: SIG_DFL, sa_mask[0]=0x00000000, sa_flags=0x00000000
SIGUSR2: [libjvm.so+0x597480], sa_mask[0]=0x00000000, sa_flags=0x10000004
SIGHUP: [libjvm.so+0x5971d0], sa_mask[0]=0x7ffbfeff, sa_flags=0x10000004
SIGINT: [libjvm.so+0x5971d0], sa_mask[0]=0x7ffbfeff, sa_flags=0x10000004
SIGTERM: [libjvm.so+0x5971d0], sa_mask[0]=0x7ffbfeff, sa_flags=0x10000004
SIGQUIT: [libjvm.so+0x5971d0], sa_mask[0]=0x7ffbfeff, sa_flags=0x10000004


---------------  S Y S T E M  ---------------

OS:Fedora release 8 (Werewolf)

uname:Linux 2.6.21.7-2.fc8xen #1 SMP Fri Feb 15 12:34:28 EST 2008 x86_64
libc:glibc 2.7 NPTL 2.7 
rlimit: STACK 10240k, CORE 0k, NPROC 61504, NOFILE 1024, AS infinity
load average:2.83 2.73 2.78

CPU:total 2 (4 cores per cpu, 1 threads per core) family 6 model 23 stepping 10, cmov, cx8, fxsr, mmx, sse, sse2, sse3, ssse3, sse4.1

Memory: 4k page, physical 7872040k(14540k free), swap 0k(0k free)

vm_info: Java HotSpot(TM) 64-Bit Server VM (14.0-b16) for linux-amd64 JRE (1.6.0_14-b08), built on May 21 2009 01:11:11 by "java_re" with gcc 3.2.2 (SuSE Lin
ux)

[error occurred during error reporting (printing date and time), id 0xb]

Answer 1

You would really need to narrow down what code it's crashing on. It looks like a JNI related issue though on the surface.

Answer 2

Fixes: - Upgrade to new JVM

Well have you tried the latest versions of the JVM?

JRE version: 6.0_14-b08

The latest release is 1.6.0_19.

Answer 3

You say you're working on "some large data set in memory," and your output includes "JavaThread 'Low Memory Detector' daemon." Are you allowing enough memory for what you're trying to do?

Answer 4

Does the segfault happen on more than one machine? If it's only one machine, it could be a hardware problem, most likely faulty RAM - try running memtest.