Jruby Rails app on Tomcat CPU Usage spikes

Question

This might also belong on serverfault. It's kind of a combo between server config and code (I think)

Here's my setup:

Rails 2.3.5 app running on jruby 1.3.1
Service Oriented backend over JMS with activeMQ 5.3 and mule 2.2.1
Tomcat 5.5 with opts: "-Xmx1536m -Xms256m -XX:MaxPermSize=256m -XX:+CMSClassUnloadingEnabled"
Java jdk 1.5.0_19
Debian Etch 4.0

Running top, every time i click a link on my site, I see my java process CPU usage spike. If it's a small page, it's sometimes just 10% usage, but sometimes on a more complicated page, my CPU goes up to 44% (never above, not sure why). In this case, a request can take upwards of minutes while my server's load average steadily climbs up to 8 or greater. This is just from clicking one link that loads a few requests from some services, nothing too complicated. The java process memory hovers around 20% most of the time.

If I leave it for a bit, load average goes back down to nothing. Clicking a few more links, climbs back up.

I'm running a small amazon instance for the rails frontend and a large instance for all the services.

Now, this is obviously unacceptable. A single user can bring spike the load average to 8 and with two people using it, it maintains that load average for the duration of our using the site. I'm wondering what I can do to inspect what's going on? I'm at a complete loss as to how I can debug this. (it doesn't happen locally when I run the rails app through jruby, not inside the tomcat container)

Can someone enlighten me as to how I might inspect on my jruby app to find out how it could possibly be using up such huge resources?

Note, I noticed this a little bit before, seemingly at random, but now, after upgrading from Rails 2.2.2 to 2.3.5 I'm seeing it ALL THE TIME and it makes the site completely unusable.

Any tips on where to look are greatly appreciated. I don't even know where to start.

Answer 1

Make sure that there is no unexpected communication between the Tomcat and something else. I would check in the first place if:

• ActiveMQ broker doesn't communicate with the other brokers in your network. By default AMQ broker start in OpenWire auto-discovery mode.
• JGroups/Multicasts in general do not communicate with something in your network.

This unnecessary load may result from the processing of the messages coming from another application.