I have a Tomcat Application Server that is configured to create a memory dump on OOM, and it is started with -Xmx1024M, so a Gigabyte should be available to him.
Now I found one such dump and it contains only 260MB of unretained memory. How is it possible that the dump is so much smaller than the size he should have available?
Only information about the usage of allocated memory will be dumped to a file. A heap dump isn't a binary image of your heap, it contains information about data types etc. and may exceed your available memory.
Permgen space is managed independently of the heap and can run out even when there's plenty of free memory overall. Some web frameworks (especially JSF) are real hogs and while easily cause the default config to run out. It can be increased with -XX:MaxPermSize=###m
Remember the system space is constrained by the sum of heap and permgen, so you can consume fewer total resources before you start getting the Cannot Create Native Thread OOM exception if you don't decrease heap by the amount PermGen is increased.