What makes hot deployment a "hard problem"?

Question

At work, we've been having a problem with "PermGen out of memory" exceptions, with the team lead deciding it was a bug in the JVM - something related to hot-deployment of code. Without explaining many details, he pointed out that hot deployment is a "hard problem", so hard that even .NET doesn't do it yet.

I found a lot of articles explaining hot deployment from the bird's-eye-view, but always lacking technical details. Could anyone point me to a technical explanation, and explain why hot deployment is "a hard problem"?

Answer 1

When a class is loaded, various static data about the class is stored in PermGen. As long as a live reference to this Class instance exists, the class instance cannot be garbage collected.

I believe that part of the problem has to do with whether or not the GC should remove old Class instances from perm gen, or not. Typically, every time you hot deploy, new class instances are added to the PermGen memory pool, and the old ones, now unused, are typically not removed. By default, the Sun JVMs will not run garbage collection in PermGen, but this can be enabled with optional "java" command arguments.

Therefore, if you hot deploy enough times, you will eventually exhaust your PermGen space.

If your web app does not shut down completely when undeployed -- if it leaves a Thread running, for example -- then all of the Class instances used by that web app will be pinned in the PermGen space. You redeploy and now have another whole copy of all of these Class instances loaded into PermGen. You undeploy and the Thread keeps going, pinning ANOTHER set of class instances in PermGen. You redeploy and load a whole net set of copies... and eventually your PermGen fills up.

You can sometimes fix this by:

• Supplying command arguments to a recent Sun JVM to enable GC in PermGen and of classes. That is: -XX:+UseConcMarkSweepGC -XX:+CMSClassUnloadingEnabled -XX:+CMSPermGenSweepingEnabled
• Using a different JVM that doesn't employ a fixed sized PermGen or that does GC on loaded classes

But this will help only if your web app shuts down completely and cleanly, leaving no live references to any of the Class instances of any Class loaded by the Class loaders for that Web App.

Even this will not necessarily fix the problem, due to class loader leaks. (As well as too many interned strings in some cases.)

Check out the following links for more (the two bolded ones have nice diagrams to illustrate part of the problem).

• Classloader leaks: the dreaded "java.lang.OutOfMemoryError: PermGen space" exception
• The Unknown Generation: Perm
• Good Riddance, PermGen OutOfMemoryError !
• Presenting the Permanent Generation
• Understanding PermGen errors (part1)
• Tomcat Wiki: How to Deal With Out Of Memory Errors

Answer 2

The problem in general terms is the security model of Java that actually attempts to prevent that a class that has already been loaded be loaded again.

Of course Java since the beginning has supported dynamic class loading, what it is difficult is class re-loading.

It was consider harmful ( and for a good reason ) that a running java application got injected with an new class with malicious code. For instance a java.lang.String cracked implementation comming from the internet , that instead of creating string, deletes some random file hile invoking the method length().

So, they way Java was conceived ( and I presume .NET CLR in consequence, because it was highly "inspired" in JVM's ) was to prevent an already loaded class to load again that same VM.

They offered a mechanism to override this "feature". Classloaders, but again the rules for the class loaders was, they should ask permission to the "parent" classloader before attempting to load a new class, if the parent has already loaded the class, the new class is ignored.

For instance I have used classloaders that load a classes from LDAP or RDBMS

The hot deploy becomes a necessity in the Java world when the application server became mainstream for Java EE ( and also create the need for micro containers like spring to avoid these kind of burden ) .

Restarting the whole app server after every compile drives anyone crazy.

So app server provider, offer this "custom" class loaders to help hot deployment, and using a configuration file, that behavior, SHOULD be disabled when set in production. But the tradeoff is you have to use tons of memory in development. So the good way to do this is restart every 3 - 4 deployments.

This doesn't happen with other languages that were designed from the beginning to load their classes.

In Ruby for instance, you can even add methods to a running class, override a method at runtime or even add a single method to an unique specific object.

The tradeoff in these kinds of environments is of course memory and speed.

I hope this helps.

EDIT

I've found this product some time ago that promises that reload is make as simple as possible. I didn't remember the link when I first wrote this answer, and I do.

It is JavaRebel from ZeroTurnaround

Answer 3

Sun JVM has PermGen space fixed, and eventually it's all consumed (yes, apparently due to a bug in classloader-related code) => OOM.

If you can use another vendor's JVM (e.g. Weblogic one), it dynamically extends PermGen space, so you'll never get permgen-related OOM.

Answer 4

Which version of java are you using? There were bugs in early Sun 1.4.2, but it's been working for a long long time.
BTW, how will you break the news to your team lead? Are you the team lead?

Answer 5

This is quite an info on Memeory outage issues. I have an issue where my Static contents throw NULL POINTER exception after HOT Deployment. Will enabling GC remove old Static Contents and re-initialize new ones. Or is there some other way to deal with this issue.

Also can we specify the order to Shut down Servlets, classes etc during hot Deployment ?

Thanks in advance Saquib