Tomcat performance issue

Question

I've got a web application that's running really slowly and occasionally hanging. It's a school-related Wicket app with reporting and editing, and also a servlet which is used by automated clients to get/post data via HTTPS.

During busy times where a lot of editing/uploading/downloading is going on, the app becomes sluggish and unresponsive. I get Wicket "PageMap still locked" errors. Tomcat seems to keep chugging along. Memory usage is managable, about 50M.

I set up YourKit to get some profiling info, and during a busy period found that 81% of the Tomcat CPU time is spent here:

org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run()
org.apache.jk.common.ChannelSocket$SocketConnection.runIt(Object[])
org.apache.jk.common.ChannelSocket.processConnection(MsgContext)
org.apache.jk.common.ChannelSocket.receive(Msg, MsgContext)
org.apache.jk.common.ChannelSocket.read(MsgContext, byte[], int, int)
java.io.BufferedInputStream.read(byte[], int, int)
java.io.BufferedInputStream.read1(byte[], int, int)
java.io.BufferedInputStream.fill()
java.net.SocketInputStream.read(byte[], int, int)
[Wall Time]  java.net.SocketInputStream.socketRead0(FileDescriptor, byte[], int, int, int)

In this app I've got about 250 clients polling the server over SSL asking for updates every 30 seconds. Most of the time, this quickly returns an empty response. Occasionally, there's a flurry of DB activity, and a whole slew of data (a few MB) may be sent back to the client.

So what's this 81% activity from? Could a few clients with really slow connections bog down tomcat threads by causing it to wait for incoming request bodies?

Anyone seen anything like this, or have any advice on how to test/troubleshoot/fix this?

Answer 1

The memory use is probably the cause.
You should check the memory use for the java process to see how much it's getting.
If it's not getting enough you have to set the Xmx jvm parameter wherever these are set for tomcat.

Answer 2

You should set up a test environment and try to reproduce this with a load test.
That way you can isolate the cause. Otherwise there are too many factors.
Also, this way you can try fixes without jeopardizing your prod env.