JVM errors after updating to Java 6 Update 14

Question

I've been getting some strange errors ( perhaps once per day ) after updating a handful of servers to run on Java 6 update 14.

The errors are similar to

#
# A fatal error has been detected by the Java Runtime Environment:
#
# java.lang.OutOfMemoryError: requested 1759920 bytes for Chunk::new. Out of swap space?
#
#  Internal Error (allocation.cpp:215), pid=26706, tid=317545360
#  Error: Chunk::new
#
# JRE version: 6.0_14-b08
# Java VM: Java HotSpot(TM) Server VM (14.0-b16 mixed mode linux-x86 )
# If you would like to submit a bug report, please visit:
#   http://java.sun.com/webapps/bugreport/crash.jsp
#

Stragely enough, the current thread is the compiler

  0x088a0800 JavaThread "CompilerThread1" daemon [_thread_blocked, id=26716, stack(0x12c7f000,0x12d00000)]
=>0x0889e400 JavaThread "CompilerThread0" daemon [_thread_in_native, id=26715, stack(0x12e55000,0x12ed6000)]

and there is more than enough memory available:

Heap

 PSYoungGen      total 256064K, used 93533K [0xa2cd0000, 0xb4290000, 0xb4290000)
  eden space 228352K, 31% used [0xa2cd0000,0xa72d6308,0xb0bd0000)
  from space 27712K, 78% used [0xb2780000,0xb3cd1150,0xb4290000)
  to   space 28032K, 0% used [0xb0bd0000,0xb0bd0000,0xb2730000)
 PSOldGen        total 2275584K, used 885858K [0x17e90000, 0xa2cd0000, 0xa2cd0000)
  object space 2275584K, 38% used [0x17e90000,0x4dfa8bf8,0xa2cd0000)
 PSPermGen       total 32128K, used 27819K [0x13e90000, 0x15df0000, 0x17e90000)
  object space 32128K, 86% used [0x13e90000,0x159bac50,0x15df0000)

I known JVM crashes are hard to debug, but I'm curious if any of you encountered similar issues - and how did you solve them.

Answer 1

The bug you are getting is not a Java code bug, but rather an issue in one of the Jit compilers. When the Jit starts up it snags a bunch of memory as memory for what it actaully does). This memory comes from the native heap.

For the really interested this error emits ultimatly from here in the VM (C++ code ahead) http://www.google.com/codesearch/p?hl=en&sa=N&cd=1&ct=rc#aRIt9pqzOVI/src/share/vm/utilities/vmError.cpp&q=OutOfMemoryError%20Out%20of%20swap%20space&l=258 unfortunatly its not a "real" OutOfMemoryError, it wont behave according to the normal rules, you cant catch it etc.

Native (JNI/JNA) methods can directly allocate memory from the OS. NIO uses memory directly, hotspot compiler, and there are others

This memory is part of the applications native heap (the stuff managed by malloc and friends), it is possible that your application has run out of native heap causing this to occur, take a look at overall memory on the box, ulimit settings etc. JNI / JNA code can also play into this somewhat, if they are able to exhaust the available memory the application has for native code. Take a look for DirectMappedBuffers from NIO as these can also steal memory off java heap.

It is quite possible given that you have just updated that you might have hit a bug inside one of the Jit compilers, GC and Jit settings can influence this so try changing the jit (-client to -server or -server to -client) to see if that has any effect, you can also try changing the GC policy (however remember that you are changing the GC policy to alter its interactions with the jit and not to fix a java memory issue).

You can also turn the Jit off completly with the command line flag -Djava.compiler=NONE, however as this removes any native code generation it will hurt your performance.

Outside of that, if you host the hprof crash file somewhere I might be able to give you some more ideas.