tags:

views:

995

answers:

7
+4  Q: 

Multiple threads stuck in native calls (Java)

I have a problem with an application running on Fedora Core 6 with JDK 1.5.0_08.

After some amount of uptime (usually some days) threads begin getting stuck in native methods.

The threads are locked in something like this:

"pool-2-thread-2571" prio=1 tid=0x08dd0b28 nid=0x319e waiting for monitor entry [0xb91fe000..0xb91ff7d4]
at java.lang.Class.getDeclaredConstructors0(Native Method)

or

"pool-2-thread-2547" prio=1 tid=0x75641620 nid=0x1745 waiting for monitor entry [0xbc7fe000..0xbc7ff554]
at sun.misc.Unsafe.defineClass(Native Method)

Especially puzzling to me is this one:

"HealthMonitor-10" daemon prio=1 tid=0x0868d1c0 nid=0x2b72 waiting for monitor entry [0xbe5ff000..0xbe5ff4d4]
at java.lang.Thread.dumpThreads(Native Method)
at java.lang.Thread.getStackTrace(Thread.java:1383)

The threads remain stuck until the VM is restarted.

Can anyone give me an idea as to what is happening here, what might be causing the native methods to block? The monitor entry address range at the top of each stuck thread is different. How can I figure out what is holding this monitor?

Any suggestions or advice will be greatly appreciated!

Thanks, David

A: 

Maybe you should use another jdk version.
For your "puzzling one", there is a bug entry for 1.5.0_08. A memory leak is reported (I do not know, if this is related to your problem):
http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=6469701

Also you could get the source code and look, what happens at line 1383. On the other side, it could just be the stack dump, after the original error occurred.

John Smithers  2008-09-01 19:22:39
A: 

@john-smithers

Unless I can prove that the problem is caused by a bug in the JDK version which was fixed later, upgrading the JDK is not an option.

I'll try looking up the source code tomorrow -- hopefully that might give me some insight into the problem. Though I do recall having a "mixed native/Java" thread dump which showed the JDK native stack traces and not being able to see anything there...

I'm running on Linux, so it doesn't look like the the bug you linked to is relevant.

abunetta  2008-09-01 20:02:44
That's a pretty short sighted attitude, JDK 1.5 is up to release 16, so while upgrading may not fix the bug you do know about, it will fix 8 releases of bugs you may not be aware of, http://java.sun.com/j2se/1.5.0/ReleaseNotes.html
Dave Cheney  2008-10-01 01:36:10
+3  A: 

My initial suspicion would be that you are experiencing some sort of class-loader realted dead lock. I imagine, that class loading needs to be synchronized at some level because class information will become available for the entire VM, not just the thread where it was initially loaded.

The fact that the methods on top of the stack are native methods seems to be pure coincidence, since part of the class loading mechanism happens to implemented that way.

I would investigate further what is going on class-loading wise. Maybe some thread uses the class loader to load a class from a network location which is slow/unavailable and thus blocks for a really long time, not yielding the monitor to other threads that want to load a class. Investigating the output when starting the JVM with -verbose:class might be one thing to try.

VoidPointer  2008-09-30 10:50:43
+1  A: 

I was having similar problems a few months ago and found the jthread(?) utility to be invaluable. You give it the process ID for your Java application and it will dump the entire stack for each thread in your process.

From the output of jthread, I could see one thread was trying to obtain a lock after having entered a monitor and another thread was trying to enter the monitor after obtaining the lock. A recipe for deadlock.

I was also wondering if your application was running into a garbage collection issue. You say it runs for a couple days before it stops like this. How long have you let it sit in the stuck state to see if maybe the GC ever finishes?

David Smith  2008-10-01 05:50:50
I think the name of the tool is jstack; I used it for thread info above. Also, sending a SIGQUIT (ctrl-\) to the JVM dumps the threads to stdout. My problem is that it seems that the deadlock is happening in code beyond my control (native code). Though I've no doubt I'm holding the lock somehow...
abunetta  2008-10-02 07:26:58
A: 

@VoidPointer

I think you are right about it being classloader related. Though the threads always block on native methods so I don't think that's a coincidence.

There are no classes that might take a long time to load (or be unavailable). The classpath includes only local jars.

Actually there is more than one classloader, each of which might be contending for some resource and becoming blocked as a result.

It seems that each time we've been able to study the threads when they are blocked one of the blocked threads has this at the top:

"pool-2-thread-2552" prio=1 tid=0x750a9ae0 nid=0x1911 waiting for monitor entry [0xbb5fd000..0xbb5ff7d4]
    at java.lang.Class.getDeclaredFields0(Native Method)
    at java.lang.Class.privateGetDeclaredFields(Class.java:2259)
    at java.lang.Class.getDeclaredFields(Class.java:1715)
    at groovy.lang.MetaClassImpl$4.run(MetaClassImpl.java:761)
    at java.security.AccessController.doPrivileged(Native Method)
    at groovy.lang.MetaClassImpl.setupProperties(MetaClassImpl.java:759)
    at groovy.lang.MetaClassImpl. (MetaClassImpl.java:160)
    at groovy.lang.MetaClassRegistry.getMetaClassFor(MetaClassRegistry.java:232)
    at groovy.lang.MetaClassRegistry.getMetaClass(MetaClassRegistry.java:139)
    - locked <0x6d9d5108> (a java.lang.Class)
    at org.codehaus.groovy.runtime.Invoker.getMetaClass(Invoker.java:105)
    at org.codehaus.groovy.runtime.InvokerHelper.getMetaClass(InvokerHelper.java:87)
    at groovy.lang.GroovyObjectSupport. (GroovyObjectSupport.java:61)
    at groovy.lang.Script. (Script.java:65)

Since the use of Groovy was removed from the application (it caused a memory leak in PermGen and wasn't really needed), we haven't seen any additional cases of the block. So this might be what was causing the problem. But I'd like to be able to at least have a definite method of reproducing before declaring it fixed.

The biggest problem here is that the lock is around a monitor entry which is in native code, seemingly pointing to a bug in the JVM. I don't know of any requirements to synchronize access to methods which use reflection and/or classloaders.

abunetta  2008-10-02 08:19:48
+1  A: 

Can you find out which thread is actually synchronizing on the monitor on which the native method is waiting? At least the thread-dump you get from the VM when you send it a SIGQUIT (kill -3) should show this information, as in

"Thread-0" prio=5 tid=0x0100b060 nid=0x84c000 waiting for monitor entry [0xb0c8a000..0xb0c8ad90]
    at Deadlock$1.run(Deadlock.java:8)
    - waiting to lock <0x255e5b38> (a java.lang.Object)
...
"main" prio=5 tid=0x01001350 nid=0xb0801000 waiting on condition [0xb07ff000..0xb0800148]
    at java.lang.Thread.sleep(Native Method)
    at Deadlock.main(Deadlock.java:21)
- locked <0x255e5b38> (a java.lang.Object)

In the dumps you've posted so far, I can't see any thread that is actually waiting to lock a specific monitor...

VoidPointer  2008-10-07 10:19:34
A: 

@VoidPointer

Unfortunately, there is no obviously deadlocking monitor. None of the values identifying monitors are repeated in a thread dump. If there were I would have noticed: I think the JVM reports suspected deadlocked threads, and there is also the [Thread Dump Analyzer][1] visualvm plugin that helps find problems in thread dumps.

I'm including a full thread dump (a little sanitized but fairly complete). I hope it gives a better indication of what is happening there.

Full thread dump Java HotSpot(TM) Client VM (1.5.0_08-b03 mixed mode):

"pool-2-thread-2571" prio=1 tid=0x08dd0b28 nid=0x319e waiting for monitor entry [0xb91fe000..0xb91ff7d4]
    at java.lang.Class.getDeclaredConstructors0(Native Method)
    at java.lang.Class.privateGetDeclaredConstructors(Class.java:2357)
    at java.lang.Class.getConstructor0(Class.java:2671)
    at java.lang.Class.newInstance0(Class.java:321)
    at java.lang.Class.newInstance(Class.java:303)
    at sun.reflect.MethodAccessorGenerator$1.run(MethodAccessorGenerator.java:381)
    at java.security.AccessController.doPrivileged(Native Method)
    at sun.reflect.MethodAccessorGenerator.generate(MethodAccessorGenerator.java:377)
    at sun.reflect.MethodAccessorGenerator.generateMethod(MethodAccessorGenerator.java:59)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:28)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:585)
    at org.apache.commons.beanutils.MethodUtils.invokeMethod(MethodUtils.java:252)
    at org.apache.commons.digester.CallMethodRule.end(CallMethodRule.java:505)
    at org.apache.commons.digester.Rule.end(Rule.java:276)
    at org.apache.commons.digester.Digester.endElement(Digester.java:1058)
    at com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.endElement(AbstractSAXParser.java:633)
    at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanEndElement(XMLDocumentFragmentScannerImpl.java:1241)
    at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl$FragmentContentDispatcher.dispatch(XMLDocumentFragmentScannerImpl.java:1685)
    at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanDocument(XMLDocumentFragmentScannerImpl.java:368)
    at com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:834)
    at com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:764)
    at com.sun.org.apache.xerces.internal.parsers.XMLParser.parse(XMLParser.java:148)
    at com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.parse(AbstractSAXParser.java:1242)
    at org.apache.commons.digester.Digester.parse(Digester.java:1586)
    ...
    at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:650)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:675)
    at java.lang.Thread.run(Thread.java:595)

"pool-2-thread-2570" prio=1 tid=0x750a6830 nid=0x2ba2 waiting for monitor entry [0xb97fe000..0xb97ff6d4]
    at java.lang.Class.getDeclaredConstructors0(Native Method)
    at java.lang.Class.privateGetDeclaredConstructors(Class.java:2357)
    at java.lang.Class.getConstructor0(Class.java:2671)
    at java.lang.Class.newInstance0(Class.java:321)
    at java.lang.Class.newInstance(Class.java:303)
    at sun.reflect.MethodAccessorGenerator$1.run(MethodAccessorGenerator.java:381)
    at java.security.AccessController.doPrivileged(Native Method)
    at sun.reflect.MethodAccessorGenerator.generate(MethodAccessorGenerator.java:377)
    at sun.reflect.MethodAccessorGenerator.generateConstructor(MethodAccessorGenerator.java:76)
    at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:30)
    at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
    at java.lang.reflect.Constructor.newInstance(Constructor.java:494)
    at java.lang.reflect.Proxy.newProxyInstance(Proxy.java:588)
    at sun.reflect.annotation.AnnotationParser.annotationForMap(AnnotationParser.java:239)
    at sun.reflect.annotation.AnnotationParser.parseAnnotation(AnnotationParser.java:229)
    at sun.reflect.annotation.AnnotationParser.parseAnnotations2(AnnotationParser.java:69)
    at sun.reflect.annotation.AnnotationParser.parseAnnotations(AnnotationParser.java:52)
    at java.lang.reflect.Field.declaredAnnotations(Field.java:1002)
    - locked <0x5322b258> (a java.lang.reflect.Field)
    at java.lang.reflect.Field.getAnnotation(Field.java:989)
    ...
    at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:650)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:675)
    at java.lang.Thread.run(Thread.java:595)

"pool-2-thread-2569" prio=1 tid=0x750a63b0 nid=0x2829 waiting for monitor entry [0xba7fe000..0xba7ff554]
    at sun.misc.Unsafe.defineClass(Native Method)
    at sun.reflect.ClassDefiner.defineClass(ClassDefiner.java:45)
    at sun.reflect.MethodAccessorGenerator$1.run(MethodAccessorGenerator.java:381)
    at java.security.AccessController.doPrivileged(Native Method)
    at sun.reflect.MethodAccessorGenerator.generate(MethodAccessorGenerator.java:377)
    at sun.reflect.MethodAccessorGenerator.generateMethod(MethodAccessorGenerator.java:59)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:28)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:585)
    at org.apache.commons.beanutils.MethodUtils.invokeMethod(MethodUtils.java:252)
    at org.apache.commons.digester.SetNextRule.end(SetNextRule.java:256)
    at org.apache.commons.digester.Rule.end(Rule.java:276)
    at org.apache.commons.digester.Digester.endElement(Digester.java:1058)
    at com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.endElement(AbstractSAXParser.java:633)
    at com.sun.org.apache.xerces.internal.parsers.AbstractXMLDocumentParser.emptyElement(AbstractXMLDocumentParser.java:221)
    at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanStartElement(XMLDocumentFragmentScannerImpl.java:872)
    at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl$FragmentContentDispatcher.dispatch(XMLDocumentFragmentScannerImpl.java:1693)
    at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanDocument(XMLDocumentFragmentScannerImpl.java:368)
    at com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:834)
    at com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:764)
    at com.sun.org.apache.xerces.internal.parsers.XMLParser.parse(XMLParser.java:148)
    at com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.parse(AbstractSAXParser.java:1242)
    at org.apache.commons.digester.Digester.parse(Digester.java:1586)
    ...
    at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:650)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:675)
    at java.lang.Thread.run(Thread.java:595)

"pool-2-thread-2568" prio=1 tid=0x750a5190 nid=0x2824 waiting for monitor entry [0xb9dfe000..0xb9dff854]
    at org.jinterop.dcom.core.JISession.addToSession(JISession.java:542)
    - waiting to lock <0x48d6a318> (a java.lang.Object)
    at org.jinterop.dcom.win32.JIComFactory.createCOMInstance(JIComFactory.java:161)
    at org.jinterop.dcom.core.JIVariant.getObjectAsComObject(JIVariant.java:1036)
    at com.company.kbapiimpl.WmiJInteropSession.executeQueryImpl(WmiJInteropSession.java:231)
    at com.company.kbapiimpl.WmiBaseSession.executeQuery(WmiBaseSession.java:74)
    at sun.reflect.GeneratedMethodAccessor61.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:585)
    ...
    at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:650)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:675)
    at java.lang.Thread.run(Thread.java:595)

"pool-2-thread-2566" prio=1 tid=0x75031348 nid=0x281f waiting for monitor entry [0xbcbfe000..0xbcbff654]
    at sun.misc.Unsafe.defineClass(Native Method)
    at sun.reflect.ClassDefiner.defineClass(ClassDefiner.java:45)
    at sun.reflect.MethodAccessorGenerator$1.run(MethodAccessorGenerator.java:381)
    at java.security.AccessController.doPrivileged(Native Method)
    at sun.reflect.MethodAccessorGenerator.generate(MethodAccessorGenerator.java:377)
    at sun.reflect.MethodAccessorGenerator.generateMethod(MethodAccessorGenerator.java:59)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:28)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:585)
    at org.apache.commons.beanutils.MethodUtils.invokeMethod(MethodUtils.java:252)
    at org.apache.commons.digester.SetNextRule.end(SetNextRule.java:256)
    at org.apache.commons.digester.Rule.end(Rule.java:276)
    at org.apache.commons.digester.Digester.endElement(Digester.java:1058)
    at com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.endElement(AbstractSAXParser.java:633)
    at com.sun.org.apache.xerces.internal.parsers.AbstractXMLDocumentParser.emptyElement(AbstractXMLDocumentParser.java:221)
    at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanStartElement(XMLDocumentFragmentScannerImpl.java:872)
    at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl$FragmentContentDispatcher.dispatch(XMLDocumentFragmentScannerImpl.java:1693)
    at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanDocument(XMLDocumentFragmentScannerImpl.java:368)
    at com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:834)
    at com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:764)
    at com.sun.org.apache.xerces.internal.parsers.XMLParser.parse(XMLParser.java:148)
    at com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.parse(AbstractSAXParser.java:1242)
    at org.apache.commons.digester.Digester.parse(Digester.java:1586)
    ...
    at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:650)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:675)
    at java.lang.Thread.run(Thread.java:595)

"pool-2-thread-2564" prio=1 tid=0x75033208 nid=0x281c waiting for monitor entry [0xbadfe000..0xbadff554]
    at sun.misc.Unsafe.defineClass(Native Method)
    at sun.reflect.ClassDefiner.defineClass(ClassDefiner.java:45)
    at sun.reflect.MethodAccessorGenerator$1.run(MethodAccessorGenerator.java:381)
    at java.security.AccessController.doPrivileged(Native Method)
    at sun.reflect.MethodAccessorGenerator.generate(MethodAccessorGenerator.java:377)
    at sun.reflect.MethodAccessorGenerator.generateMethod(MethodAccessorGenerator.java:59)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:28)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:585)
    at org.apache.commons.beanutils.MethodUtils.invokeMethod(MethodUtils.java:252)
    at org.apache.commons.digester.SetNextRule.end(SetNextRule.java:256)
    at org.apache.commons.digester.Rule.end(Rule.java:276)
    at org.apache.commons.digester.Digester.endElement(Digester.java:1058)
    at com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.endElement(AbstractSAXParser.java:633)
    at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanEndElement(XMLDocumentFragmentScannerImpl.java:1241)
    at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl$FragmentContentDispatcher.dispatch(XMLDocumentFragmentScannerImpl.java:1685)
    at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanDocument(XMLDocumentFragmentScannerImpl.java:368)
    at com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:834)
    at com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:764)
    at com.sun.org.apache.xerces.internal.parsers.XMLParser.parse(XMLParser.java:148)
    at com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.parse(AbstractSAXParser.java:1242)
    at org.apache.commons.digester.Digester.parse(Digester.java:1586)
    ...
    at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:650)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:675)
    at java.lang.Thread.run(Thread.java:595)

"pool-2-thread-2562" prio=1 tid=0x750368a8 nid=0x2817 waiting for monitor entry [0xbd5fe000..0xbd5ff754]
    at sun.misc.Unsafe.defineClass(Native Method)
    at sun.reflect.ClassDefiner.defineClass(ClassDefiner.java:45)
    at sun.reflect.MethodAccessorGenerator$1.run(MethodAccessorGenerator.java:381)
    at java.security.AccessController.doPrivileged(Native Method)
    at sun.reflect.MethodAccessorGenerator.generate(MethodAccessorGenerator.java:377)
    at sun.reflect.MethodAccessorGenerator.generateMethod(MethodAccessorGenerator.java:59)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:28)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:585)
    at org.apache.commons.beanutils.MethodUtils.invokeMethod(MethodUtils.java:252)
    at org.apache.commons.digester.SetNextRule.end(SetNextRule.java:256)
    at org.apache.commons.digester.Rule.end(Rule.java:276)
    at org.apache.commons.digester.Digester.endElement(Digester.java:1058)
    at com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.endElement(AbstractSAXParser.java:633)
    at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanEndElement(XMLDocumentFragmentScannerImpl.java:1241)
    at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl$FragmentContentDispatcher.dispatch(XMLDocumentFragmentScannerImpl.java:1685)
    at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanDocument(XMLDocumentFragmentScannerImpl.java:368)
    at com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:834)
    at com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:764)
    at com.sun.org.apache.xerces.internal.parsers.XMLParser.parse(XMLParser.java:148)
    at com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.parse(AbstractSAXParser.java:1242)
    at org.apache.commons.digester.Digester.parse(Digester.java:1586)
    ...
    at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:650)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:675)
    at java.lang.Thread.run(Thread.java:595)

"pool-2-thread-2561" prio=1 tid=0x759f7cd8 nid=0x20c4 waiting for monitor entry [0xbb3fe000..0xbb3ff754]
    at sun.misc.Unsafe.defineClass(Native Method)
    at sun.reflect.ClassDefiner.defineClass(ClassDefiner.java:45)
    at sun.reflect.MethodAccessorGenerator$1.run(MethodAccessorGenerator.java:381)
    at java.security.AccessController.doPrivileged(Native Method)
    at sun.reflect.MethodAccessorGenerator.generate(MethodAccessorGenerator.java:377)
    at sun.reflect.MethodAccessorGenerator.generateMethod(MethodAccessorGenerator.java:59)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:28)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:585)
    at org.apache.commons.beanutils.MethodUtils.invokeMethod(MethodUtils.java:252)
    at org.apache.commons.digester.CallMethodRule.end(CallMethodRule.java:505)
    at org.apache.commons.digester.Rule.end(Rule.java:276)
    at org.apache.commons.digester.Digester.endElement(Digester.java:1058)
    at com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.endElement(AbstractSAXParser.java:633)
    at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanEndElement(XMLDocumentFragmentScannerImpl.java:1241)
    at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl$FragmentContentDispatcher.dispatch(XMLDocumentFragmentScannerImpl.java:1685)
    at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanDocument(XMLDocumentFragmentScannerImpl.java:368)
    at com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:834)
    at com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:764)
    at com.sun.org.apache.xerces.internal.parsers.XMLParser.parse(XMLParser.java:148)
    at com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.parse(AbstractSAXParser.java:1242)
    at org.apache.commons.digester.Digester.parse(Digester.java:1586)
    ...
    at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:650)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:675)
    at java.lang.Thread.run(Thread.java:595)

"pool-2-thread-2557" prio=1 tid=0x75037460 nid=0x1936 waiting for monitor entry [0xb9bfe000..0xb9bff5d4]
    at java.lang.Class.getDeclaredMethods0(Native Method)
    at java.lang.Class.privateGetDeclaredMethods(Class.java:2395)
    at java.lang.Class.getDeclaredMethod(Class.java:1907)
    at java.io.ObjectStreamClass.getPrivateMethod(ObjectStreamClass.java:1354)
    at java.io.ObjectStreamClass.access$1700(ObjectStreamClass.java:52)
    at java.io.ObjectStreamClass$2.run(ObjectStreamClass.java:421)
    at java.security.AccessController.doPrivileged(Native Method)
    at java.io.ObjectStreamClass. (ObjectStreamClass.java:400)
    at java.io.ObjectStreamClass.lookup(ObjectStreamClass.java:297)
    at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1035)
    at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1375)
    at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1347)
    at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1290)
    at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1079)
    at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:302)
    at java.util.ArrayList.writeObject(ArrayList.java:569)
    at sun.reflect.GeneratedMethodAccessor2115.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:585)
    at java.io.ObjectStreamClass.invokeWriteObject(ObjectStreamClass.java:917)
    at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1339)
    at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1290)
    at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1079)
    at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1375)
    at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1347)
    at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1290)
    at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1079)
    at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1375)
    at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1347)
    at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1290)
    at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1079)
    at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1375)
    at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1347)
    at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1290)
    at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1079)
    at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:302)
    ...
    - locked <0x52ffb6e8> (a com.company.CompressibleObject)
    at com.company.AClass.compress(AClass.java:448)
    ...
    at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:650)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:675)
    at java.lang.Thread.run(Thread.java:595)

"pool-2-thread-2554" prio=1 tid=0x750aa318 nid=0x1931 waiting for monitor entry [0xba1fe000..0xba1ff554]
    at sun.misc.Unsafe.defineClass(Native Method)
    at sun.reflect.ClassDefiner.defineClass(ClassDefiner.java:45)
    at sun.reflect.MethodAccessorGenerator$1.run(MethodAccessorGenerator.java:381)
    at java.security.AccessController.doPrivileged(Native Method)
    at sun.reflect.MethodAccessorGenerator.generate(MethodAccessorGenerator.java:377)
    at sun.reflect.MethodAccessorGenerator.generateMethod(MethodAccessorGenerator.java:59)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:28)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:585)
    at org.apache.commons.beanutils.MethodUtils.invokeMethod(MethodUtils.java:252)
    at org.apache.commons.digester.CallMethodRule.end(CallMethodRule.java:505)
    at org.apache.commons.digester.Rule.end(Rule.java:276)
    at org.apache.commons.digester.Digester.endElement(Digester.java:1058)
    at com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.endElement(AbstractSAXParser.java:633)
    at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanEndElement(XMLDocumentFragmentScannerImpl.java:1241)
    at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl$FragmentContentDispatcher.dispatch(XMLDocumentFragmentScannerImpl.java:1685)
    at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanDocument(XMLDocumentFragmentScannerImpl.java:368)
    at com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:834)
    at com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:764)
    at com.sun.org.apache.xerces.internal.parsers.XMLParser.parse(XMLParser.java:148)
    at com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.parse(AbstractSAXParser.java:1242)
    at org.apache.commons.digester.Digester.parse(Digester.java:1586)
    ...
    at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:650)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:675)
    at java.lang.Thread.run(Thread.java:595)

"pool-2-thread-2552" prio=1 tid=0x750a9ae0 nid=0x1911 waiting for monitor entry [0xbb5fd000..0xbb5ff7d4]
    at java.lang.Class.getDeclaredFields0(Native Method)
    at java.lang.Class.privateGetDeclaredFields(Class.java:2259)
    at java.lang.Class.getDeclaredFields(Class.java:1715)
    at groovy.lang.MetaClassImpl$4.run(MetaClassImpl.java:761)
    at java.security.AccessController.doPrivileged(Native Method)
    at groovy.lang.MetaClassImpl.setupProperties(MetaClassImpl.java:759)
    at groovy.lang.MetaClassImpl. (MetaClassImpl.java:160)
    at groovy.lang.MetaClassRegistry.getMetaClassFor(MetaClassRegistry.java:232)
    at groovy.lang.MetaClassRegistry.getMetaClass(MetaClassRegistry.java:139)
    - locked <0x6d9d5108> (a java.lang.Class)
    at org.codehaus.groovy.runtime.Invoker.getMetaClass(Invoker.java:105)
    at org.codehaus.groovy.runtime.InvokerHelper.getMetaClass(InvokerHelper.java:87)
    at groovy.lang.GroovyObjectSupport. (GroovyObjectSupport.java:61)
    at groovy.lang.Script. (Script.java:65)
    at return_new_com.company. (return_new_com.company.FileNotFoundProbingException_groups_1___)
    at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
    at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
    at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
    at java.lang.reflect.Constructor.newInstance(Constructor.java:494)
    at java.lang.Class.newInstance0(Class.java:350)
    at java.lang.Class.newInstance(Class.java:303)
    at org.codehaus.groovy.runtime.InvokerHelper.createScript(InvokerHelper.java:529)
    at groovy.lang.GroovyShell.parse(GroovyShell.java:526)
    at groovy.lang.GroovyShell.parse(GroovyShell.java:506)
    at groovy.lang.GroovyShell.evaluate(GroovyShell.java:484)
    at groovy.lang.GroovyShell.evaluate(GroovyShell.java:426)
    at org.codehaus.groovy.bsf.GroovyEngine.eval(GroovyEngine.java:144)
    at org.apache.bsf.BSFManager$5.run(BSFManager.java:445)
    at java.security.AccessController.doPrivileged(Native Method)
    at org.apache.bsf.BSFManager.eval(BSFManager.java:442)
    ...
    at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:650)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:675)
    at java.lang.Thread.run(Thread.java:595)

"pool-2-thread-2551" prio=1 tid=0x7553d268 nid=0x18fb waiting for monitor entry [0xba5fe000..0xba5ff754]
    at org.jinterop.dcom.core.JISession.addToSession(JISession.java:542)
    - waiting to lock <0x48d6a318> (a java.lang.Object)
    at org.jinterop.dcom.win32.JIComFactory.createCOMInstance(JIComFactory.java:161)
    at org.jinterop.dcom.core.JIVariant.getObjectAsComObject(JIVariant.java:1036)
    ...
    at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:650)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:675)
    at java.lang.Thread.run(Thread.java:595)

"pool-2-thread-2547" prio=1 tid=0x75641620 nid=0x1745 waiting for monitor entry [0xbc7fe000..0xbc7ff554]
    at sun.misc.Unsafe.defineClass(Native Method)
    at sun.reflect.ClassDefiner.defineClass(ClassDefiner.java:45)
    at sun.reflect.MethodAccessorGenerator$1.run(MethodAccessorGenerator.java:381)
    at java.security.AccessController.doPrivileged(Native Method)
    at sun.reflect.MethodAccessorGenerator.generate(MethodAccessorGenerator.java:377)
    at sun.reflect.MethodAccessorGenerator.generateMethod(MethodAccessorGenerator.java:59)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:28)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:585)
    at org.apache.commons.beanutils.MethodUtils.invokeMethod(MethodUtils.java:252)
    at org.apache.commons.digester.SetNextRule.end(SetNextRule.java:256)
    at org.apache.commons.digester.Rule.end(Rule.java:276)
    at org.apache.commons.digester.Digester.endElement(Digester.java:1058)
    at com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.endElement(AbstractSAXParser.java:633)
    at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanEndElement(XMLDocumentFragmentScannerImpl.java:1241)
    at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl$FragmentContentDispatcher.dispatch(XMLDocumentFragmentScannerImpl.java:1685)
    at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanDocument(XMLDocumentFragmentScannerImpl.java:368)
    at com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:834)
    at com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:764)
    at com.sun.org.apache.xerces.internal.parsers.XMLParser.parse(XMLParser.java:148)
    at com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.parse(AbstractSAXParser.java:1242)
    at org.apache.commons.digester.Digester.parse(Digester.java:1586)
    ...
    at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:650)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:675)
    at java.lang.Thread.run(Thread.java:595)

"pool-2-thread-2546" prio=1 tid=0x7567c988 nid=0x1743 waiting for monitor entry [0xbc9ff000..0xbc9ff7d4]
    at org.jinterop.dcom.core.JIComOxidRuntime.addUpdateOXIDs(JIComOxidRuntime.java:276)
    - waiting to lock <0x48d6a278> (a java.lang.Object)
    at org.jinterop.dcom.core.JISession.addToSession(JISession.java:563)
    at org.jinterop.dcom.core.JISession.addToSession(JISession.java:546)
    at org.jinterop.dcom.win32.JIComFactory.createCOMInstance(JIComFactory.java:161)
    at org.jinterop.dcom.core.JIVariant.getObjectAsComObject(JIVariant.java:1036)
    ...
    at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:650)
    at java.util.concurrent.
abunetta  2008-10-07 16:17:37

related questions

Java Time Zone is messed up
Eclipse on win64
Automate builds for Java RCP for deployment with JNLP
Why are professors or schools picking Java over C++ to teach to students?
Is there a real benefit of using J#?
Public/Popular Websites using JavaServer Faces
Why can't I use a try block around my super() call?
Accessing post variables using Java Servlets
Personal Linux web server
Is this really widening vs autoboxing?
How can I Java webstart multiple, dependent, native libraries?
Why can't I call toString() on a Java primitive?
How do I use Java to read from a file that is actively being written?
What code analysis tools do you use for your Java projects?
IllegalArgumentException or NullPointerException for a null parameter?
How do I configure and communicate with a serial port?
What is the best way to parse strings in Java
Getting started with a custom JXTA PeerGroup
Creating a custom button in Java
How to get started "writing" a code coverage tool?
Which Build-/Configuration Management Tool?
What is the difference between an int and an Integer in Java/C#?
What is the meaning of the type safety warning in certain Java generics casts?
How would you access Object properties from within an object method?
Converting CSV File to XML in Java