We're running a small web application written JRuby on Rails running under Tomcat. We're using a Spring back-end that's shared with another production web application. Unfortunately, we keep running into PermGen problems.
OS: Ubuntu Linux 2.6.24-24-server #1 SMP x86_64 GNU/Linux Java: 1.6.0_21 Tomcat: 6.0.28 JRuby: 1.5.0 Rails: 2.3.7
We're currently getting crawled by Google, Yahoo and Baidu, so site usage is up. I've been monitoring Tomcat with JConsole and we're definitely seeing a problem with an excessive number of classes. When tomcat launches, we have about 12,000 classes loaded. After 8 hours, we have almost 75,000 classes loaded. PermGen goes from 100MB to 460MB in the same time.
Class unloading is working, but it only unloaded ~500 classes over that same 8 hour period. PermGen never seems to get collected.
We're running with the following VM options for Tomcat:
-Xms2048m -Xmx2048m -XX:MaxPermSize=512m -XX:PermSize=128m \
-XX:+UseConcMarkSweepGC -XX:+UseParNewGC -XX:ParallelGCThreads=4 \
-XX:+CMSClassUnloadingEnabled -XX:+CMSPermGenSweepingEnabled
Obviously there's some kind of leak. The question is how where? Any advice on how to track down who and what is responsible for this? I'm hoping it's some really silly mistake on our part, but I'm not sure where to begin.
Any advice would be greatly appreciated.
EDIT
It looks like we're seeing one new class created for every single incoming request.
EDIT 2
It's definitely related to JRuby. Using JConsole, I enabled Verbose mode for the class loader. Here's a sample from catalina.out:
[Loaded anon_class1275113147_895127379 from file:/opt/apache-tomcat-6.0.28/webapps/notes/WEB-INF/lib/jruby-core-1.5.0.jar]
[Loaded anon_class1354333392_895127376 from file:/opt/apache-tomcat-6.0.28/webapps/notes/WEB-INF/lib/jruby-core-1.5.0.jar]
[Loaded anon_class1402528430_895127373 from file:/opt/apache-tomcat-6.0.28/webapps/notes/WEB-INF/lib/jruby-core-1.5.0.jar]
So the question becomes how do I track down the party responsible for creating those extra classes?
EDIT 3
Not sure if this the problem, but somehow we're ending up with an insane number of class loaders. Ran jmap -permstat PID
and got:
class_loader classes bytes parent_loader alive? type
total = 1320 135748 947431296 N/A alive=1, dead=1319 N/A
That seems a little excessive. The majority are one of three kinds of classloaders: sun.reflect.DelegatingClassLoader
, org.jruby.util.JRubyClassLoader
or org.jruby.util.ClassCache$OneShotClassLoader
. Again, sample output from jmap -permstat
:
class_loader classes bytes parent_loader alive? type
0x00007f71f4e93d58 1 3128 0x00007f71f4d54680 dead sun/reflect/DelegatingClassLoader@0x00007f72ef9a6dc0
0x00007f721e51e2a0 57103 316038936 0x00007f720431c958 dead org/jruby/util/JRubyClassLoader@0x00007f72f2fd1158
0x00007f72182f2b10 4 12944 0x00007f721d7f3030 dead org/jruby/util/JRubyClassLoader@0x00007f72f2fd1158
0x00007f721d7d50d8 9 457520 0x00007f720431c958 dead org/jruby/util/ClassCache$OneShotClassLoader@0x00007f72f3ce2368