I want to limit the maximum memory used by the JVM. Note, this is not just the heap, I want to limit the total memory used by this process.
You shouldn't have to worry about the stack leaking memory (it is highly uncommon). The only time you can have the stack get out of control is with infinite (or really deep) recursion.
This is just the heap. Sorry, didn't read your question fully at first.
You need to run the JVM with the following command line argument.
-Xmx<ammount of memory>
Example:
-Xmx1024m
That will allow a max of 1GB of memory for the JVM.
use the arguments -Xms -Xmx M, G after the numbers for indicating Megs and Gigs of bytes respectively. -Xms indicates the minimum and -Xmx the maximum
The answer above is kind of correct, you can't gracefully control how much native memory a java process allocates. It depends on what your application is doing.
That said, depending on platform, you may be able to do use some mechanism, ulimit for example, to limit the size of a java or any other process.
Just don't expect it to fail gracefully if it hits that limit. Native memory allocation failures are much harder to handle than allocation failures on the java heap. There's a fairly good chance the application will crash but depending on how critical it is to the system to keep the process size down that might still suit you.
I've come across this statement reading the following article.
http://java.dzone.com/articles/java-performance-tuning?page=0,1
"The NativeHeap can be increasded by -XX:MaxDirectMemorySize=256M (default is 128)"
I've never used it. Maybe you'll find is useful. The spelling error is not mine.