Java OutOfMemoryError due to Linux RAM disk cache not freed

Question

The process will run fine all day, then, bam, without warning, it will throw this error. Sometimes seemingly in the middle of doing nothing. It will happen at seemingly random times during the day. I checked to see if anything else was running on the machine, like scheduled backups or something, but found nothing. The machine has enough physical memory (2GB, with about 1GB free for a 3-500MB load), and has sufficient -Xmx specified.

According to our sysadmin, the problem is that the RAM that the kernel uses as a disk cache (apparently all but 8MB) is not freed when the JVM needs to allocate memory, so the JVM process throws an OutOfMemoryError. This could be because Java asks the kernel if enough memory is available before allocating and finds that it is insufficient, resulting in a crash.

I would like to think, however, that Java simply tries to allocate the memory via the kernel, and when the kernel gets such a request, it makes room for the application by throwing our some of the disk cache.

Has anyone else run in to the issue, and if so, what was the error, and how did you solve it? We are currently using jdk1.6.0_20 on SLES 10 SP2 Linux 2.6.16.60-0.42.9-smp in VMWare ESX.

Answer 1

Do you know how much memory java have allocated when throwing that exception? The problem might be that Java have used all the memory specified with -Xmx. (Remember that -Mmx gives the maximum allowed memory usage, so java will throw an OutOfMemmoryException if it's usage grows above the specified size, even if the linux kernel would have been able to give it more memmory).

But to find out if this is the case, try to give it a -Xms equal to the size specified with -Xms. (-Ms set the initial size, and java should newer go below this, so it will allocate all needed memory at startup).

If java still starts up, and still give you OutOfMemoryException then the problem is most likely that your -Mx is to small.

ps: I am pretty sure that a memory allocation can't fail with a normal linux kernel, due to overcommit by the kernel so if you really are out of memory for the entire system, you would get page faults(And the oom killer, would kill a task) not java exceptions. But java may use some special features to allocate memory so I am not sure if that apply here.

Answer 2

According to our sysadmin, the problem is that the RAM that the kernel uses as a disk cache (apparently all but 8MB) is not freed when the JVM needs to allocate memory, so the JVM process throws an OutOfMemoryError.

I think that your sysadmin has got this wrong. Linux will automatically and transparently grow and shrink the size of the buffer cache depending on the system's need for physical memory for running processes. If it doesn't then there is something seriously broken with your OS.

It is more likely that your Java application has a memory leak that is causing it to use a steadily increasing amount of heap space. Eventually you run against the -Xmx limit and get an OOME. I would start investigating this by running the application with GC logging enabled. This will tell you how the heap is growing over time, how much space the GC is reclaiming and so on.

Another possibility is that you don't have enough swap space configured.