Excessive memory allocation in Java Sandbox security

Question

Under the Java security model it is possible to block most dangerous actions from untrusted classes, but the last time I checked (a few years ago now) it was still possible for untrusted code to perform a denial of service attack by continually allocating memory until the JVM crashes with an OutOfMemoryException. Looking now, I can't see any improvement in the situation.

I have a requirement to run untrusted code from 3rd parties inside a Java application and I'd like to know if it is possible to somehow restrict the heap/stack space that a class or thread can allocate in the Java security model. Thus preventing memory allocation based DoS attacks. I know about -Xss, but as I understand it that restricts all threads, most of which need no restriction.

I have also considered creating a container for the untrusted code that will run in its own JVM and communicate with the main app through sockets, or doing some static analysis on the untrusted code. However, these both sound like more effort than I hoped, although if someone knows of a trick or opensource library for this I'm interested.

So, is there a way to restrict the amount of memory than a thread can allocate to itself or some other way of preventing memory allocation denial of service attacks in Java?

Answer 1

There is currently no way to do this with standard APIs in Java.

More people have been interested in this and there is a JSR underway for this called Resource Consumption Management API which may be something to look into.

Answer 2

You will need to run the untrusted code in a separate process. There may still be ways to DoS, for instance on old versions of Windows you could easily use up all GDI resources (not tried recently, not now we have Swing).