Trying to locate a leak! What does anon mean for pmap?

Question

I'm trying to locate where my memory has gone for a java process running in linux. Someone suggested I use pmap -x to see exactly what the memory is doing.

The output is really long but basically a good portion of it is a repeat of this:

00007fbf75f6a000    1016       -       -       - rwx--    [ anon ]
00007fbf76068000      12       -       -       - -----    [ anon ]

What exactly does this mean? Why do I have so many entries of this (4000+)?

Answer 1

See this part of System Performance Tuning for anonymous memory.

Answer 2

Something to understand about JVM memory characteristics: They are very greedy. It is pretty standard to include in the JVM startup the starting memory the process should take and the maximum memory that the process can take. However, what Java then tends to do (and this behavior will vary between version of the JVM) is consume the available memory until it is full, and then garbage collect. It may garbage collect earlier, but it won't be as aggressive, because it has more memory available. When it does the "stop the world" garbage collection once it has hit max memory it can typically free up a lot of memory (50% is not unusual) however, the jvm doesn't give that memory back to the operating system. It just holds on to it until it needs it.

There are some who claim that this isn't an issue because the operating system will simply put that memory in swap space on disk and it will not get in the way if the java process doesn't need it. That has not been my observation. First, until the big garbage collection, the memory isn't really free, and can be quite fractured, making it impossible to effectively swap it away. And second, even after the garbage collection, I have seen at least on windows real OS memory starvation because of the JVM.

See this question for some advice on how to reclaim memory for the OS in such a scenario. I don't know much about pmap, but it is likely extra memory that the JVM either used and freed, or requested from the OS because it was running low but hasn't used yet. (The comment suggests I really don't know anything about pmap, which is entirely true, but I stand by the rest of the answer in regard to the underlying problem of "where my memory has gone for a java process running in linux.")

Answer 3

Anon blocks are "large" blocks allocated via malloc or mmap -- see the manpages. As such, they have nothing to do with the Java heap (other than the fact that the entire heap should be stored in just such a block).

In my experience, thread stacks also use anon blocks. If you see a lot of anon blocks that all have the same size, and that size is 512k to 4Mb (the example below is repeated over a dozen times for a Tomcat process that I have running), that's the likely cause. Depending on the program, you may have up to a few dozen of these; if you're seeing thousands, it means you have a problem with threading.

b089f000    504K rwx--    [ anon ]
b091d000     12K -----    [ anon ]
b0920000    504K rwx--    [ anon ]
b099e000     12K -----    [ anon ]
b09a1000    504K rwx--    [ anon ]
b0a1f000     12K -----    [ anon ]

But that leaves a question: why are you using pmap to diagnose a Java memory issue?

Answer 4

Use Eclipse MAT (when you get OutOfMemoryExceptions in the Java Heap not the native heap).