What are some tools that can analyse memory usage outside of the heap in Java?

Question

We have weird memory leak problem with a Java process running in linux has an ever growing swap usage. So naturally we looked at the heap dump and also used a profiler to monitor it over a period of time. We found that

1) The number of threads does not grow
2) The heap usage does not grow
3) Yet the (VIRT) usage keeps growing (which can become a problem because the system starts to run out of swap space)

Now there are a ton of tools that can dump the heap or monitor the heap but none for memory outside of the heap. Anyone have any ideas?

PS this is a remote server, we don't have access to any GUI.

Answer 1

Doesn't the case where 1) the process's heap space does not change but 2) the swap usage does change indicate that some other process on the box might be responsible for sudden growths in memory usage?

In other words, my understanding was that something like swap usage was regulated by the OS - so if a Java process's own heap usage does not change but the swap usage does, that would seem to indicate to me that the problem lies elsewhere, and it just so happens that the OS is choosing your Java process to start eating up swap space.

Or do I have the wrong understanding on swap space?

Answer 2

You could be leaking something in native memory, like Sockets. Are there lots of connections happening, and are you closing out the connections in a finally block?

Answer 3

Do the other parts of JVM memory grow? For example the permgen space?

Do you use native libraries (JNI)?

Answer 4

I'll try to answer by answering another question. Is it possible that the heap size configuration of the JVM is more than free physical memory you have? Even if you define initial heap size much smaller than maximum heap size, and JVM allocates it all, it will never returns it back to OS, even if you garbage collect it all, and you don't have any allocations anymore. Don't confiugre 1.5GB max heap on 1G RAM server. Please check that configured maximum heap size "enters" the free RAM you have, together with other processes, especially if it's a server application. Otherwise, your application will get a lot of page faults and will swap all the time.