How to detect Out Of Memory condition?

Question

I have an application running on Websphere Application Server 6.0 and it crashes nearly every day because of Out-Of-Memory. From verbose GC is certain there are the memory leaks(many of them)

Unfortunately the application is provided by external vendor and getting things fixed is slow & painful process. As part of the process I need to gather the logs and heapdumps each time the OOM occurs.

Now I'm looking for some way how to automate it. Fundamental problem is how to detect OOM condition. One way would be to create shell script which will periodically search for new heapdumps. This approach seems me a kinda dirty. Another approach might be to leverage the JMX somehow. But I have little or no experience in this area and don't have much idea how to do it.

Or is in WAS some kind of trigger/hooks for this? Thank you very much for every advice!

Answer 1

You can pass the following arguments to the JVM on startup and a heap dump will be automatically generated on an OutOfMemoryError. The second argument lets you specify the path for the heap dump file. By using this at least you could check for the existence of a specific file to see if a heap dump has occurred.

-XX:+HeapDumpOnOutOfMemoryError
-XX:HeapDumpPath=<value>

Answer 2

And alternative to waiting until the application has crashed may be to script a controlled restart like every night if you're optimistic that it can survive for twelve hours..

Maybe even websphere can do that for you !?

Answer 3

You could add a listener (Session scoped or Application scope attribute listener) class that would be called each time a new object is added in session/app scope.

In this - you can attempt to check the total memory used by app (Log it) as as call run gc (note that invoking it will not imply gc will always run)

(The above is for the logging part and gc based on usage growth)

For scheduled gc: In addition you can keep a timer task class that runs after every few hrs and does a request for gc.

Answer 4

Have you looked at JConsole ? It uses JMX to give you visibility of a variety of JVM metrics, including memory info. It would probably be worth monitoring your application using this to begin with, to get a feel for how/when the memory is consumed. You may find the memory is consumed uniformly over the day, or when using certain features.

Take a look at the detecting low memory section of the above link.

If you need you can then write a JMX client to watch the application automatically and trigger whatever actions required. JConsole will indicate which JMX methods you need to poll.

Answer 5

I see two options if you want heap dumping automated but @Mark's solution with heap dump on OOM isn't satisfactory.

1. You can use the MemoryMXBean to detect high memory pressure, and then programmatically create a heap dump if the usage (or usage delta) seems high.
2. You can periodically get memory usage info and generate heap dumps with a cron'd shell script using jmap (works both locally and remote).

It would be nice if you could have a callback on OOM, but, uhm, that callback probably would just crash with an OOM error. :)

Answer 6

Have you had a look on the jvisualvm tool in the latest Java 6 JDK's?

It is great for inspecting running code.

Answer 7

I'd dispute that the you need the heap dumps when the OOM occurs. Periodic gathering of the information over time should give the picture of what's going on.

As has been observed various tools exist for analysing these problems. I have had success with ITCAM for WebSphere, as an IBMer I have ready access to that. We were very quickly able to indentify the exact lines of code in out problem situation.

If there's any way you can get a tool of that nature then that's the way to go.

Answer 8

Our experience with ITCAM has been less than stellar from the monitoring perspective. We dumped it in favor of CA Wily Introscope.