Memory leak in JBoss

Question

I am having a very strange behavior in JBoss, and I'd like avail myself of the Collective Wisdom of the SO Crowd.

We're using JBoss (4.0.4 I think) to serve SOAP calls. In fact, it's used as glorified RPC server, no more. We're running out of memory when we have 20+ clients sending their requests at the same time. The requests consist of the incoming rather small request (proper SOAP) and the returning result packet that is essentially one long SOAP string (and the contents of the string are XML). Yes I realize this is suboptimal. Don't ask.

I've traced the leak to an instance of org.jboss.axis.message.SAX2EventRecorder that hold 4 million objects (strings and Integers). Now, even the longest response does not carry 4MB of data. The requests are all smaller than 40K. Something is fishy there, but I can't find any documentation on the Web.

Can someone tell me what the recorder is used for? And how do I get rid of it? Or may be configure it to be less memory-hungry? Any help is appreciated.

---

Update: To clarify - I did do memory dump, and the dump shows an array or 4,000,000+ objects, Strings and Integers. The array is owned by a org.jboss.axis.message.SAX2EventRecorder which is in turn held by these guys:

org.jboss.axis.message.SOAPEnvelopeAxisImpl@0x19c31fd8 (141 bytes) : field recorder org.jboss.axis.message.RPCParamElementImpl@0x19c32260 (123 bytes) : field recorder org.jboss.axis.message.SOAPBodyAxisImpl@0x19c32160 (121 bytes) : field recorder org.jboss.axis.message.RPCElement@0x19c321e0 (124 bytes) : field recorder org.jboss.axis.encoding.DeserializationContextImpl@0x19c332f0 (67 bytes) : field recorder org.jboss.axis.message.SAX2EventRecorder$objArrayVector@0x19c33398 (24 bytes) : field this$0

The data structures of our own app are quete bloated, but not to this degree.

Another update: powers that be have found a "powers-that-be-solution": we're switching to 64-bit memory. Hurray.

Answer 1

Run with the JVM arg -XX:-HeapDumpOnOutOfMemoryError. This will give you heap dumps when you run out of memory. You can then analyze the heap dump with the jhat tool (it comes with your JDK). Alternatively, you can use the jconsole tool (which also comes with your JDK) to request a heap dump at any time using the memory management MBeans.

It will tell you what those 4 million objects actually contain which might give you insight into why the software isn't releasing that memory.

EDIT:

It seems you're not the only one with this issue. There are 2 bug reports filed with Axis

AXIS-2698

AXIS-2749

See also AXIS-1771, it has some interesting information regarding deserialization and ways to mediate its impact. Which version of Axis are you using?

Answer 2

If you can, run the application under Java 6. The latest version includes VisualVM for profiling. It should be able to show the growth of memory. You can attach to a Java 5 VM, but it doesn't show as much.