JVM Thread dumps containing monitors without locking threads

Question

What could be the cause of JVM thread dumps that show threads waiting to lock on a monitor, but the monitors do not have corresponding locking threads?

Java 1.5_14 on Windows 2003

Answer 1

That's just a wild guess, but could it be, that a thread locks itself by trying to acquire a lock twice? Probably it would help if you could post some code.

Answer 2

Java's synchronized blocks are reentrant so I would not think this should be a problem, Although I am not ruling something like this out.

Are you thinking this is just an issue with the way the thread dump is reporting things?

The strange thing to me, is that the blocked threads are waiting (per the format of the thread dump) on a particular lockId, but there is no thread in the thread dump that holds this lock.

Answer 3

Here is an example thread dump:

Full thread dump Java HotSpot(TM) Server VM (1.5.0_11-b03 mixed mode):

"http-0.0.0.0-8080-47" daemon prio=6 tid=0x4cfcf408 nid=0xb94 runnable [0x556ef000..0x556efb9c] at java.net.SocketInputStream.socketRead0(Native Method) at java.net.SocketInputStream.read(SocketInputStream.java:129) at org.apache.coyote.http11.InternalInputBuffer.fill(InternalInputBuffer.java:737) at org.apache.coyote.http11.InternalInputBuffer.parseRequestLine(InternalInputBuffer.java:398) at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:827) at org.apache.coyote.http11.Http11BaseProtocol$Http11ConnectionHandler.processConnection(Http11BaseProtocol.java:664) at org.apache.tomcat.util.net.PoolTcpEndpoint.processSocket(PoolTcpEndpoint.java:527) at org.apache.tomcat.util.net.MasterSlaveWorkerThread.run(MasterSlaveWorkerThread.java:112) at java.lang.Thread.run(Thread.java:595)

"http-0.0.0.0-8080-46" daemon prio=6 tid=0x4cf77408 nid=0x8d8 waiting for monitor entry [0x555ae000..0x555afc1c] at com.mypackage.content.FolderAccessCacheManager.getCache(FolderAccessCacheManager.java:445) - waiting to lock <0x0c6e68d8> (a com.mypackage.content.FolderAccessCacheManager) at com.mypackage.content.FolderAccessCacheManager.getFolderAccessNode(FolderAccessCacheManager.java:117) at com.mypackage.content.FolderAccessNode.checkParentFolderAccess(FolderAccessNode.java:359) at com.mypackage.content.FolderAccessNode.checkParentFolderAccess(FolderAccessNode.java:373) at com.mypackage.content.FolderAccessNode.checkAccess(FolderAccessNode.java:342) at com.mypackage.content.FolderAccessNode.canAccess(FolderAccessNode.java:266) at com.mypackage.content.FolderAccessNode.checkParentFolderAccess(FolderAccessNode.java:364) at com.mypackage.content.FolderAccessNode.checkAccess(FolderAccessNode.java:342) at com.mypackage.content.FolderAccessNode.canAccess(FolderAccessNode.java:252) at com.mypackage.content.FolderAccessNode.checkParentFolderAccess(FolderAccessNode.java:364) at com.mypackage.content.FolderAccessNode.checkAccess(FolderAccessNode.java:342) at com.mypackage.content.FolderAccessNode.canAccess(FolderAccessNode.java:259) at com.mypackage.content.FolderAccessNode.checkParentFolderAccess(FolderAccessNode.java:364) at com.mypackage.content.FolderAccessNode.checkParentFolderAccess(FolderAccessNode.java:373) at com.mypackage.content.FolderAccessNode.checkAccess(FolderAccessNode.java:342) at com.mypackage.content.FolderAccessNode.canAccess(FolderAccessNode.java:252) at com.mypackage.content.FolderAccessNode.checkParentFolderAccess(FolderAccessNode.java:364) at com.mypackage.content.FolderAccessNode.checkAccess(FolderAccessNode.java:342) at com.mypackage.content.FolderAccessNode.canAccess(FolderAccessNode.java:252) at com.mypackage.content.FolderAccessController.loadAccessableFoldersByRules(FolderAccessController.java:556) at com.mypackage.content.FolderAccessController.loadAccessableFolders(FolderAccessController.java:495) at com.mypackage.content.FolderAccessController.getAccessableFolderIDs(FolderAccessController.java:185) at com.mypackage.content.ContentAccessController.getContentAccessCriteria(ContentAccessController.java:212) at com.mypackage.util.ServletSessionUtil.initSession(ServletSessionUtil.java:84) at com.mypackage.sso.Gatekeeper.initSession(Gatekeeper.java:210) at com.mypackage.sso.AbstractGatekeeper.doFilter(AbstractGatekeeper.java:149) at com.mypackage.sso.Gatekeeper.doFilter(Gatekeeper.java:114) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:202) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:173) at com.mypackage.mvc.servlet.CharacterEncodingFilter.doFilter(CharacterEncodingFilter.java:55) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:202) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:173) at org.jboss.web.tomcat.filters.ReplyHeaderFilter.doFilter(ReplyHeaderFilter.java:96) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:202) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:173) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:213) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:178) at org.jboss.web.tomcat.security.SecurityAssociationValve.invoke(SecurityAssociationValve.java:175) at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:432) at org.jboss.web.tomcat.security.JaccContextValve.invoke(JaccContextValve.java:74) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:126) at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:105) at org.jboss.web.tomcat.tc5.jca.CachedConnectionValve.invoke(CachedConnectionValve.java:156) at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:541) at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:107) at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:148) at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:869) at org.apache.coyote.http11.Http11BaseProtocol$Http11ConnectionHandler.processConnection(Http11BaseProtocol.java:664) at org.apache.tomcat.util.net.PoolTcpEndpoint.processSocket(PoolTcpEndpoint.java:527) at org.apache.tomcat.util.net.MasterSlaveWorkerThread.run(MasterSlaveWorkerThread.java:112) at java.lang.Thread.run(Thread.java:595)

"http-0.0.0.0-8080-45" daemon prio=6 tid=0x4cb88db8 nid=0xe00 waiting for monitor entry [0x5556f000..0x5556fc9c] at com.mypackage.content.FolderAccessCacheManager.getCache(FolderAccessCacheManager.java:445) - waiting to lock <0x0c6e68d8> (a com.mypackage.content.FolderAccessCacheManager) at com.mypackage.content.FolderAccessCacheManager.getFolderAccessNode(FolderAccessCacheManager.java:117) at com.mypackage.content.FolderAccessNode.checkParentFolderAccess(FolderAccessNode.java:359) at com.mypackage.content.FolderAccessNode.checkParentFolderAccess(FolderAccessNode.java:373) at com.mypackage.content.FolderAccessNode.checkAccess(FolderAccessNode.java:342) at com.mypackage.content.FolderAccessNode.canAccess(FolderAccessNode.java:252) at com.mypackage.content.FolderAccessController.loadAccessableFoldersByRules(FolderAccessController.java:556) at com.mypackage.content.FolderAccessController.loadAccessableFolders(FolderAccessController.java:495) at com.mypackage.content.FolderAccessController.getAccessableFolderIDs(FolderAccessController.java:185) at com.mypackage.content.ContentAccessController.getContentAccessCriteria(ContentAccessController.java:184) at com.mypackage.util.ServletSessionUtil.initSession(ServletSessionUtil.java:84) at com.mypackage.util.ServletSessionUtil.initGuestSession(ServletSessionUtil.java:147) at com.mypackage.sso.Gatekeeper.initGuestSession(Gatekeeper.java:255) at com.mypackage.sso.AbstractGatekeeper.doFilter(AbstractGatekeeper.java:128) at com.mypackage.sso.Gatekeeper.doFilter(Gatekeeper.java:114) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:202) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:173) at com.mypackage.mvc.servlet.CharacterEncodingFilter.doFilter(CharacterEncodingFilter.java:55) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:202) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:173) at org.jboss.web.tomcat.filters.ReplyHeaderFilter.doFilter(ReplyHeaderFilter.java:96) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:202) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:173) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:213) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:178) at org.jboss.web.tomcat.security.SecurityAssociationValve.invoke(SecurityAssociationValve.java:175) at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:432) at org.jboss.web.tomcat.security.JaccContextValve.invoke(JaccContextValve.java:74) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:126) at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:105) at org.jboss.web.tomcat.tc5.jca.CachedConnectionValve.invoke(CachedConnectionValve.java:156) at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:541) at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:107) at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:148) at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:869) at org.apache.coyote.http11.Http11BaseProtocol$Http11ConnectionHandler.processConnection(Http11BaseProtocol.java:664) at org.apache.tomcat.util.net.PoolTcpEndpoint.processSocket(PoolTcpEndpoint.java:527) at org.apache.tomcat.util.net.MasterSlaveWorkerThread.run(MasterSlaveWorkerThread.java:112) at java.lang.Thread.run(Thread.java:595)

"http-0.0.0.0-8080-44" daemon prio=6 tid=0x4cb88c30 nid=0xd40 runnable [0x5552f000..0x5552fd1c] at java.net.SocketInputStream.socketRead0(Native Method) at java.net.SocketInputStream.read(SocketInputStream.java:129) at org.apache.coyote.http11.InternalInputBuffer.fill(InternalInputBuffer.java:737) at org.apache.coyote.http11.InternalInputBuffer.parseRequestLine(InternalInputBuffer.java:398) at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:827) at org.apache.coyote.http11.Http11BaseProtocol$Http11ConnectionHandler.processConnection(Http11BaseProtocol.java:664) at org.apache.tomcat.util.net.PoolTcpEndpoint.processSocket(PoolTcpEndpoint.java:527) at org.apache.tomcat.util.net.MasterSlaveWorkerThread.run(MasterSlaveWorkerThread.java:112) at java.lang.Thread.run(Thread.java:595)

"http-0.0.0.0-8080-43" daemon prio=6 tid=0x4cb88aa8 nid=0x768 waiting for monitor entry [0x554ef000..0x554efd9c] at com.mypackage.content.FolderAccessCacheManager.getCache(FolderAccessCacheManager.java:445) - waiting to lock <0x0c6e68d8> (a com.mypackage.content.FolderAccessCacheManager) at com.mypackage.content.FolderAccessCacheManager.getFolderAccessNode(FolderAccessCacheManager.java:117) at com.mypackage.content.FolderAccessNode.checkParentFolderAccess(FolderAccessNode.java:359) at com.mypackage.content.FolderAccessNode.checkParentFolderAccess(FolderAccessNode.java:373) at com.mypackage.content.FolderAccessNode.checkParentFolderAccess(FolderAccessNode.java:373) at com.mypackage.content.FolderAccessNode.checkAccess(FolderAccessNode.java:342) at com.mypackage.content.FolderAccessNode.canAccess(FolderAccessNode.java:252) at com.mypackage.content.FolderAccessNode.checkParentFolderAccess(FolderAccessNode.java:364) at com.mypackage.content.FolderAccessNode.checkAccess(FolderAccessNode.java:342) at com.mypackage.content.FolderAccessNode.canAccess(FolderAccessNode.java:259) at com.mypackage.content.FolderAccessNode.checkParentFolderAccess(FolderAccessNode.java:364) at com.mypackage.content.FolderAccessNode.checkAccess(FolderAccessNode.java:342) at com.mypackage.content.FolderAccessNode.canAccess(FolderAccessNode.java:252) at com.mypackage.content.FolderAccessNode.checkParentFolderAccess(FolderAccessNode.java:364) at com.mypackage.content.FolderAccessNode.checkParentFolderAccess(FolderAccessNode.java:373) at com.mypackage.content.FolderAccessNode.checkAccess(FolderAccessNode.java:342) at com.mypackage.content.FolderAccessNode.canAccess(FolderAccessNode.java:252) at com.mypackage.content.FolderAccessNode.checkParentFolderAccess(FolderAccessNode.java:364) at com.mypackage.content.FolderAccessNode.checkAccess(FolderAccessNode.java:342) at com.mypackage.content.FolderAccessNode.canAccess(FolderAccessNode.java:252) at com.mypackage.content.FolderAccessController.loadAccessableFoldersByRules(FolderAccessController.java:556) at com.mypackage.content.FolderAccessController.loadAccessableFolders(FolderAccessController.java:495) at com.mypackage.content.FolderAccessController.getAccessableFolderIDs(FolderAccessController.java:185) at com.mypackage.content.ContentAccessController.getContentAccessCriteria(ContentAccessController.java:212) at com.mypackage.util.ServletSessionUtil.initSession(ServletSessionUtil.java:84) at com.mypackage.sso.Gatekeeper.initSession(Gatekeeper.java:210) at com.mypackage.sso.AbstractGatekeeper.doFilter(AbstractGatekeeper.java:149) at com.mypackage.sso.Gatekeeper.doFilter(Gatekeeper.java:114) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:202) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:173) at com.mypackage.mvc.servlet.CharacterEncodingFilter.doFilter(CharacterEncodingFilter.java:55) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:202) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:173) at org.jboss.web.tomcat.filters.ReplyHeaderFilter.doFilter(ReplyHeaderFilter.java:96) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:202) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:173) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:213) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:178) at org.jboss.web.tomcat.security.SecurityAssociationValve.invoke(SecurityAssociationValve.java:175) at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:432) at org.jboss.web.tomcat.security.JaccContextValve.invoke(JaccContextValve.java:74) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:126) at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:105) at org.jboss.web.tomcat.tc5.jca.CachedConnectionValve.invoke(CachedConnectionValve.java:156) at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:541) at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:107) at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:148) at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:869) at org.apache.coyote.http11.Http11BaseProtocol$Http11ConnectionHandler.processConnection(Http11BaseProtocol.java:664) at org.apache.tomcat.util.net.PoolTcpEndpoint.processSocket(PoolTcpEndpoint.java:527) at org.apache.tomcat.util.net.MasterSlaveWorkerThread.run(MasterSlaveWorkerThread.java:112) at java.lang.Thread.run(Thread.java:595)

"http-0.0.0.0-8080-42" daemon prio=6 tid=0x4cddc230 nid=0xb74 waiting for monitor entry [0x554ae000..0x554afa1c] at com.mypackage.content.FolderAccessCacheManager.getCache(FolderAccessCacheManager.java:445) - waiting to lock <0x0c6e68d8> (a com.mypackage.content.FolderAccessCacheManager) at com.mypackage.content.FolderAccessCacheManager.getFolderAccessNode(FolderAccessCacheManager.java:117) at com.mypackage.content.FolderAccessNode.checkParentFolderAccess(FolderAccessNode.java:359) at com.mypackage.content.FolderAccessNode.checkAccess(FolderAccessNode.java:342) at com.mypackage.content.FolderAccessNode.canAccess(FolderAccessNode.java:252) at com.mypackage.content.FolderAccessNode.checkParentFolderAccess(FolderAccessNode.java:364) at com.mypackage.content.FolderAccessNode.checkAccess(FolderAccessNode.java:342) at com.mypackage.content.FolderAccessNode.canAccess(FolderAccessNode.java:252) at com.mypackage.content.FolderAccessNode.checkParentFolderAccess(FolderAccessNode.java:364) at com.mypackage.content.FolderAccessNode.checkAccess(FolderAccessNode.java:342) at com.mypackage.content.FolderAccessNode.canAccess(FolderAccessNode.java:252) at com.mypackage.content.FolderAccessNode.checkParentFolderAccess(FolderAccessNode.java:364) at com.mypackage.content.FolderAccessNode.checkAccess(FolderAccessNode.java:342) at com.mypackage.content.FolderAccessNode.canAccess(FolderAccessNode.java:259) at com.mypackage.content.FolderAccessController.loadAccessableFoldersByRules(FolderAccessController.java:556) at com.mypackage.content.FolderAccessController.loadAccessableFolders(FolderAccessController.java:495) at com.mypackage.content.FolderAccessController.getAccessableFolderIDs(FolderAccessController.java:185) at com.mypackage.content.ContentAccessController.getContentAccessCriteria(ContentAccessController.java:212) at com.mypackage.util.ServletSessionUtil.initSession(ServletSessionUtil.java:84) at com.mypackage.sso.Gatekeeper.initSession(Gatekeeper.java:210) at com.mypackage.sso.AbstractGatekeeper.doFilter(AbstractGatekeeper.java:149) at com.mypackage.sso.Gatekeeper.doFilter(Gatekeeper.java:114) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:202) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:173) at com.mypackage.mvc.servlet.CharacterEncodingFilter.doFilter(CharacterEncodingFilter.java:55) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:202) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:173) at org.jboss.web.tomcat.filters.ReplyHeaderFilter.doFilter(ReplyHeaderFilter.java:96) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:202) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:173) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:213) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:178) at org.jboss.web.tomcat.security.SecurityAssociationValve.invoke(SecurityAssociationValve.java:175) at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:432) at org.jboss.web.tomcat.security.JaccContextValve.invoke(JaccContextValve.java:74) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:126) at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:105) at org.jboss.web.tomcat.tc5.jca.CachedConnectionValve.invoke(CachedConnectionValve.java:156) at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:541) at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:107) at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:148) at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:869) at org.apache.coyote.http11.Http11BaseProtocol$Http11ConnectionHandler.processConnection(Http11BaseProtocol.java:664) at org.apache.tomcat.util.net.PoolTcpEndpoint.processSocket(PoolTcpEndpoint.java:527) at org.apache.tomcat.util.net.MasterSlaveWorkerThread.run(MasterSlaveWorkerThread.java:112) at java.lang.Thread.run(Thread.java:595)

"http-0.0.0.0-8080-41" daemon prio=6 tid=0x4cddc008 nid=0x774 waiting for monitor entry [0x5546e000..0x5546fa9c] at com.mypackage.mvc.servlet.Dispatcher.service(Dispatcher.java:224) - waiting to lock <0x0e0c4ec0> (a com.mypackage.mvc.RequestProcessor) at javax.servlet.http.HttpServlet.service(HttpServlet.java:810) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:252) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:173) at org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDispatcher.java:672) at org.apache.catalina.core.ApplicationDispatcher.processRequest(ApplicationDispatcher.java:463) at org.apache.catalina.core.ApplicationDispatcher.doForward(ApplicationDispatcher.java:398) at org.apache.catalina.core.ApplicationDispatcher.forward(ApplicationDispatcher.java:301) at org.apache.jsp.main_jsp._jspService(main_jsp.java:84) at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:97) at javax.servlet.http.HttpServlet.service(HttpServlet.java:810) at org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:334) at org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:314) at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:264) at javax.servlet.http.HttpServlet.service(HttpServlet.java:810) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:252) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:173) at com.mypackage.sso.AbstractGatekeeper.doFilter(AbstractGatekeeper.java:166) at com.mypackage.sso.Gatekeeper.doFilter(Gatekeeper.java:114) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:202) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:173) at com.mypackage.mvc.servlet.CharacterEncodingFilter.doFilter(CharacterEncodingFilter.java:55) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:202) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:173) at org.jboss.web.tomcat.filters.ReplyHeaderFilter.doFilter(ReplyHeaderFilter.java:96) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:202) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:173) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:213) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:178) at org.jboss.web.tomcat.security.SecurityAssociationValve.invoke(SecurityAssociationValve.java:175) at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:432) at org.jboss.web.tomcat.security.JaccContextValve.invoke(JaccContextValve.java:74) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:126) at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:105) at org.jboss.web.tomcat.tc5.jca.CachedConnectionValve.invoke(CachedConnectionValve.java:156) at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:541) at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:107) at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:148) at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:869) at org.apache.coyote.http11.Http11BaseProtocol$Http11ConnectionHandler.processConnection(Http11BaseProtocol.java:664) at org.apache.tomcat.util.net.PoolTcpEndpoint.processSocket(PoolTcpEndpoint.java:527) at org.apache.tomcat.util.net.MasterSlaveWorkerThread.run(MasterSlaveWorkerThread.java:112) at java.lang.Thread.run(Thread.java:595)

"http-0.0.0.0-8080-40" daemon prio=6 tid=0x4cb611d8 nid=0xc78 waiting for monitor entry [0x5542e000..0x5542fb1c] at com.mypackage.content.FolderAccessCacheManager.getCache(FolderAccessCacheManager.java:445) - waiting to lock <0x0c6e68d8> (a com.mypackage.content.FolderAccessCacheManager) at com.mypackage.content.FolderAccessCacheManager.getFolderAccessNode(FolderAccessCacheManager.java:117) at com.mypackage.content.FolderAccessNode.checkParentFolderAccess(FolderAccessNode.java:359) at com.mypackage.content.FolderAccessNode.checkParentFolderAccess(FolderAccessNode.java:373) at com.mypackage.content.FolderAccessNode.checkParentFolderAccess(FolderAccessNode.java:373) at com.mypackage.content.FolderAccessNode.checkAccess(FolderAccessNode.java:342) at com.mypackage.content.FolderAccessNode.canAccess(FolderAccessNode.java:252) at com.mypackage.content.FolderAccessNode.checkParentFolderAccess(FolderAccessNode.java:364) at com.mypackage.content.FolderAccessNode.checkParentFolderAccess(FolderAccessNode.java:373) at com.mypackage.content.FolderAccessNode.checkAccess(FolderAccessNode.java:342) at com.mypackage.content.FolderAccessNode.canAccess(FolderAccessNode.java:252) at com.mypackage.content.FolderAccessNode.checkParentFolderAccess(FolderAccessNode.java:364) at com.mypackage.content.FolderAccessNode.checkParentFolderAccess(FolderAccessNode.java:373) at com.mypackage.content.FolderAccessNode.checkAccess(FolderAccessNode.java:342) at com.mypackage.content.FolderAccessNode.canAccess(FolderAccessNode.java:252) at com.mypackage.content.FolderAccessController.loadAccessableFoldersByRules(FolderAccessController.java:556) at com.mypackage.content.FolderAccessController.loadAccessableFolders(FolderAccessController.java:495) at com.mypackage.content.FolderAccessController.getAccessableFolderIDs(FolderAccessController.java:185) at com.mypackage.content.ContentAccessController.getContentAccessCriteria(ContentAccessController.java:184) at com.mypackage.util.ServletSessionUtil.initSession(ServletSessionUtil.java:84) at com.mypackage.sso.Gatekeeper.initSession(Gatekeeper.java:210) at com.mypackage.sso.AbstractGatekeeper.doFilter(AbstractGatekeeper.java:149) at com.mypackage.sso.Gatekeeper.doFilter(Gatekeeper.java:114) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:202) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:173) at com.mypackage.mvc.servlet.CharacterEncodingFilter.doFilter(CharacterEncodingFilter.java:55) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:202) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:173) at org.jboss.web.tomcat.filters.ReplyHeaderFilter.doFilter(ReplyHeaderFilter.java:96) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:202) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:173) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:213) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:178) at org.jboss.web.tomcat.security.SecurityAssociationValve.invoke(SecurityAssociationValve.java:175) at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:432) at org.jboss.web.tomcat.security.JaccContextValve.invoke(JaccContextValve.java:74) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:126) at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:105) at org.jboss.web.tomcat.tc5.jca.CachedConnectionValve.invoke(CachedConnectionValve.java:156) at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:541) at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:107) at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:148) at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:869) at org.apache.coyote.http11.Http11BaseProtocol$Http11ConnectionHandler.processConnection(Http11BaseProtocol.java:664) at org.apache.tomcat.util.net.PoolTcpEndpoint.processSocket(PoolTcpEndpoint.java:527) at org.apache.tomcat.util.net.MasterSlaveWorkerThread.run(MasterSlaveWorkerThread.java:112) at java.lang.Thread.run(Thread.java:595)

"http-0.0.0.0-8080-39" daemon prio=6 tid=0x4c7fd068 nid=0xff0 in Object.wait() [0x553ef000..0x553efb9c] at java.lang.Object.wait(Native Method) - waiting on <0x0e340dc8> (a org.apache.tomcat.util.net.MasterSlaveWorkerThread) at java.lang.Object.wait(Object.java:474) at org.apache.tomcat.util.net.MasterSlaveWorkerThread.await(MasterSlaveWorkerThread.java:81) - locked <0x0e340dc8> (a org.apache.tomcat.util.net.MasterSlaveWorkerThread) at org.apache.tomcat.util.net.MasterSlaveWorkerThread.run(MasterSlaveWorkerThread.java:107) at java.lang.Thread.run(Thread.java:595)

"http-0.0.0.0-8080-38" daemon prio=6 tid=0x4d186008 nid=0xeb4 in Object.wait() [0x553af000..0x553afc1c] at java.lang.Object.wait(Native Method) - waiting on <0x0e340da0> (a org.apache.tomcat.util.net.MasterSlaveWorkerThread) at java.lang.Object.wait(Object.java:474) at org.apache.tomcat.util.net.MasterSlaveWorkerThread.await(MasterSlaveWorkerThread.java:81) - locked <0x0e340da0> (a org.apache.tomcat.util.net.MasterSlaveWorkerThread) at org.apache.tomcat.util.net.MasterSlaveWorkerThread.run(MasterSlaveWorkerThread.java:107) at java.lang.Thread.run(Thread.java:595)

"http-0.0.0.0-8080-37" daemon prio=6 tid=0x0078d0b0 nid=0xc8c runnable [0x5536f000..0x5536fc9c] at java.net.SocketInputStream.socketRead0(Native Method) at java.net.SocketInputStream.read(SocketInputStream.java:129) at org.apache.coyote.http11.InternalInputBuffer.fill(InternalInputBuffer.java:737) at org.apache.coyote.http11.InternalInputBuffer.parseRequestLine(InternalInputBuffer.java:398) at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:827) at org.apache.coyote.http11.Http11BaseProtocol$Http11ConnectionHandler.processConnection(Http11BaseProtocol.java:664) at org.apache.tomcat.util.net.PoolTcpEndpoint.processSocket(PoolTcpEndpoint.java:527) at org.apache.tomcat.util.net.MasterSlaveWorkerThread.run(MasterSlaveWorkerThread.java:112) at java.lang.Thread.run(Thread.java:595)

"http-0.0.0.0-8080-36" daemon prio=6 tid=0x4ce41008 nid=0xc64 waiting for monitor entry [0x5532f000..0x5532fd1c] at

Answer 4

Yes normally each monitor which is locked must have an owner Thread. Maybe your stack dump was not complete (too long) or maybe the dumping was not consistent. I could imagine that it is not stopping the world, so a locked monitor is dumped but the thread who owns the lock releases it before beeing dumped (this is just an guess).

Can you some where upload the dump as a text file for easier searching, and tell us which monitor you are looking at.

Answer 5

Here is the uploaded thread dump: http://drop.io/6zwhfkn

This is behavior the routinely and repeatedly happens so I am having a hard time believing that the locking thread has released the lock while the thread dump is being taken.

The lock in question is: <0x0c6e68d8>, you will notice that many threads are waiting for this lock.

If you use the Thread Dump Analyzer, there is a section for these types of monitors, that do not have locking threads (which makes me think there is a concrete answer for why/how this happens)

Any help, ideas or suggestions are appreciated.

Answer 6

Does your code by any change use any JNI? (i.e. are you running any native code launched from Java?).

We've seen a similar behavior, but JDK 1.6.0_05. App appears to deadlock, but Jstack shows threads waiting for a lock that no other threads are holding onto. We have some JNI code, so it's possible we're corrupting something.

We haven't found a solution for this and the issue is only reproducible on 1 machine.

Answer 7

Do those waiting threads wait for ever, or do they eventually proceed?

If the latter, it may be that the lock is held by the garbage collector.

You can add the arguments -verbose:gc with -XX:+PrintGCDetails on your java command line to be told when GCs are occurring. If gc activity coincides with your slowdowns it may indicate that this is the problem.

Here's some information on garbage collection.

Answer 8

Hi Kevin, Did you ever get to the bottom of the problem? I also encountered the exact same problem on a CentOS box with 4CPUs, 2Gb of RAM, running JRE 1.5.0_09.

The monitor that has no locking thread appears in 3 different places in my application, 2 of which are in the Sun JDK code and 1 in my application code. The lock in my application code is a ReentrantLock, the other two are normal synchronized{} blocks.

It might be just a coincidence, but in all 3 cases the lock object was a static field in the class. Does that ring a bell to anyone? Could classloaders and GC have anything to do with it?

Answer 9

I had a similar problem today, and it also involved accesses of static resources.

The short version is that a class made GUI changes in a static block, and outside of the AWT-EventQueue thread, which were blocked by the AWT TreeLock, then the EventQueue made a reference to the blocked class, which forced it to wait on the class loader's monitor for that class.

The key observation here is that the lock for the class loader did not show up as locked in the thread dump.

The full answer can be found on this thread.

Answer 10

Have you tried upgrading to Java 1.6? A bug could be your issue if you're only on 1.5.