We have a tomcat application that started crashing every 10-15 minutes after we upgraded the OS ("aptitude dist-upgrade" of the system, which is a 32-bit Debian Lenny). java didn't get upgraded. A Full thread dump is generated in catalina.out when jvm crashes. But there is no error/exception showing in any of the threads. Does anyone know what might have caused the problem? Thanks.
Here is the beginning of thread dump:
Mar 6, 2010 7:00:02 AM org.apache.coyote.http11.Http11BaseProtocol pause
INFO: Pausing Coyote HTTP/1.1 on http-8080
Full thread dump Java HotSpot(TM) Server VM (1.5.0_17-b04 mixed mode):
"Thread-27" prio=1 tid=0x082f5aa8 nid=0xe95 waiting on condition [0xa1bfc000..0xa1bfce30]
at java.lang.Thread.sleep(Native Method)
at org.apache.catalina.core.StandardService.stop(StandardService.java:495)
at org.apache.catalina.core.StandardServer.stop(StandardServer.java:734)
at org.apache.catalina.startup.Catalina.stop(Catalina.java:602)
at org.apache.catalina.startup.Catalina$CatalinaShutdownHook.run(Catalina.java:645)
"SIGHUP handler" daemon prio=1 tid=0xa4ce85d0 nid=0xe93 in Object.wait() [0xa1cfe000..0xa1cfef30]
at java.lang.Object.wait(Native Method)
- waiting on (a org.apache.catalina.startup.Catalina$CatalinaShutdownHook)
at java.lang.Thread.join(Thread.java:1095)
- locked (a org.apache.catalina.startup.Catalina$CatalinaShutdownHook)
at java.lang.Thread.join(Thread.java:1148)
at java.lang.Shutdown.runHooks(Shutdown.java:132)
at java.lang.Shutdown.sequence(Shutdown.java:173)
at java.lang.Shutdown.exit(Shutdown.java:218)
- locked (a java.lang.Class)
at java.lang.Terminator$1.handle(Terminator.java:35)
at sun.misc.Signal$1.run(Signal.java:195)
at java.lang.Thread.run(Thread.java:595)
"http-8080-Monitor" prio=1 tid=0xa3ff1780 nid=0xbb4 in Object.wait() [0xa1eea000..0xa1eeafb0]
at java.lang.Object.wait(Native Method)
- waiting on (a org.apache.tomcat.util.threads.ThreadPool$MonitorRunnable)
at org.apache.tomcat.util.threads.ThreadPool$MonitorRunnable.run(ThreadPool.java:564)
- locked (a org.apache.tomcat.util.threads.ThreadPool$MonitorRunnable)
at java.lang.Thread.run(Thread.java:595)
"http-8080-Processor25" daemon prio=1 tid=0xa3ff0aa8 nid=0xbb3 sleeping[0xa1f6b000..0xa1f6c030]
at java.lang.Thread.sleep(Native Method)
at org.apache.tomcat.util.net.LeaderFollowerWorkerThread.runIt(LeaderFollowerWorkerThread.java:62)
at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:689)
at java.lang.Thread.run(Thread.java:595)
.....
Following are what got updated in the upgraded
[UPGRADE] base-files 5lenny4 -> 5lenny5 [UPGRADE] bind9-host 1:9.5.1.dfsg.P3-1 -> 1:9.5.1.dfsg.P3-1+lenny1 [UPGRADE] dnsutils 1:9.5.1.dfsg.P3-1 -> 1:9.5.1.dfsg.P3-1+lenny1 [UPGRADE] dpkg 1.14.25 -> 1.14.28 [UPGRADE] dselect 1.14.25 -> 1.14.28 [UPGRADE] filter-syslog 1.22-1 -> 1.23-1 [UPGRADE] gzip 1.3.12-6 -> 1.3.12-6+lenny1 [UPGRADE] krb5-clients 1.6.dfsg.4~beta1-5lenny1 -> 1.6.dfsg.4~beta1-5lenny2 [UPGRADE] krb5-rsh-server 1.6.dfsg.4~beta1-5lenny1 -> 1.6.dfsg.4~beta1-5lenny2 [UPGRADE] krb5-user 1.6.dfsg.4~beta1-5lenny1 -> 1.6.dfsg.4~beta1-5lenny2 [UPGRADE] libbind9-40 1:9.5.1.dfsg.P3-1 -> 1:9.5.1.dfsg.P3-1+lenny1 [UPGRADE] libc6 2.7-18 -> 2.7-18lenny2 [UPGRADE] libc6-dev 2.7-18 -> 2.7-18lenny2 [UPGRADE] libc6-i686 2.7-18 -> 2.7-18lenny2 [UPGRADE] libdns45 1:9.5.1.dfsg.P3-1 -> 1:9.5.1.dfsg.P3-1+lenny1 [UPGRADE] libexpat1 2.0.1-4+lenny1 -> 2.0.1-4+lenny3 [UPGRADE] libisc45 1:9.5.1.dfsg.P3-1 -> 1:9.5.1.dfsg.P3-1+lenny1 [UPGRADE] libisccc40 1:9.5.1.dfsg.P3-1 -> 1:9.5.1.dfsg.P3-1+lenny1 [UPGRADE] libisccfg40 1:9.5.1.dfsg.P3-1 -> 1:9.5.1.dfsg.P3-1+lenny1 [UPGRADE] libkadm55 1.6.dfsg.4~beta1-5lenny1 -> 1.6.dfsg.4~beta1-5lenny2 [UPGRADE] libkrb53 1.6.dfsg.4~beta1-5lenny1 -> 1.6.dfsg.4~beta1-5lenny2 [UPGRADE] libltdl3 1.5.26-4 -> 1.5.26-4+lenny1 [UPGRADE] liblwres40 1:9.5.1.dfsg.P3-1 -> 1:9.5.1.dfsg.P3-1+lenny1 [UPGRADE] libmysqlclient15off 5.0.51a-24+lenny2 -> 5.0.51a-24+lenny3 [UPGRADE] libpq5 8.3.8-0lenny1 -> 8.3.9-0lenny1 [UPGRADE] libssl0.9.8 0.9.8g-15+lenny5 -> 0.9.8g-15+lenny6 [UPGRADE] libxerces2-java 2.9.1-2 -> 2.9.1-2+lenny1 [UPGRADE] linux-image-2.6.26-2-686 2.6.26-19lenny2 -> 2.6.26-21lenny3 [UPGRADE] linux-libc-dev 2.6.26-19lenny2 -> 2.6.26-21lenny3 [UPGRADE] locales 2.7-18 -> 2.7-18lenny2 [UPGRADE] login 1:4.1.1-6 -> 1:4.1.1-6+lenny1 [UPGRADE] munin-node 1.2.6-10~lenny1 -> 1.2.6-10~lenny2 [UPGRADE] mysql-common 5.0.51a-24+lenny2 -> 5.0.51a-24+lenny3 [UPGRADE] openssl 0.9.8g-15+lenny5 -> 0.9.8g-15+lenny6 [UPGRADE] passwd 1:4.1.1-6 -> 1:4.1.1-6+lenny1 [UPGRADE] python2.5 2.5.2-15 -> 2.5.2-15+lenny1 [UPGRADE] python2.5-minimal 2.5.2-15 -> 2.5.2-15+lenny1 [UPGRADE] sudo 1.6.9p17-2 -> 1.6.9p17-2+lenny1 [UPGRADE] tzdata 2009l-0lenny1 -> 2010a-0lenny1 [UPGRADE] usbutils 0.73-10 -> 0.73-10lenny1 [UPGRADE] wallet-client 0.9-1 -> 0.10-1~sbp50+1