Our application connects to several jmx agents (about 50) to poll jmx data in frequent interval. After some time (about a day) application becomes irresponsive. We looked at thread dump and seeing large number of threads stuck on ClientNotifForwarder..
Eg:
"ClientNotifForwarder-50" daemon prio=3 tid=0x09aa3800 nid=0x330 runnable [0xd3a90000..0xd3a90c70] java.lang.Thread.State: RUNNABLE at java.net.SocketInputStream.socketRead0(Native Method) at java.net.SocketInputStream.read(SocketInputStream.java:129) at java.io.BufferedInputStream.fill(BufferedInputStream.java:218) at java.io.BufferedInputStream.read(BufferedInputStream.java:237) - locked <0xe2538af0> (a java.io.BufferedInputStream) at java.io.DataInputStream.readByte(DataInputStream.java:248) at sun.rmi.transport.StreamRemoteCall.executeCall(StreamRemoteCall.java:195) at sun.rmi.server.UnicastRef.invoke(UnicastRef.java:142) at com.sun.jmx.remote.internal.PRef.invoke(Unknown Source) at javax.management.remote.rmi.RMIConnectionImpl_Stub.fetchNotifications(Unknown Source) at javax.management.remote.rmi.RMIConnector$RMINotifClient.fetchNotifs(RMIConnector.java:1291) at com.sun.jmx.remote.internal.ClientNotifForwarder$NotifFetcher.fetchNotifs(ClientNotifForwarder.java:503) at com.sun.jmx.remote.internal.ClientNotifForwarder$NotifFetcher.run(ClientNotifForwarder.java:395) at com.sun.jmx.remote.internal.ClientNotifForwarder$LinearExecutor$1.run(ClientNotifForwarder.java:83)
Any help in identifying root cause is appreciated...
Thanks, Raja