What is the proper way to reload your application in Tomcat from Ant without encountering Permgen Exceptions?

Question

My team and I are developing a Java web app using Tomcat as our development testing server and Ant as our build tool. Our platform is Windows but we're running Tomcat out of Cygwin as our preferred shell is bash. We've run into consistent problems with running out of memory due to the frequent restarts of the application in order to make newly implemented features available for testing.

We were originally trying to reload the application in place without restarting the server using the supplied Catalina ant task. That worked for a few iterations but then we'd always run out of Permgen space. To be clear, this is referring to the Ant Tasks bundled with Tomcat that start, stop, deploy, undeploy, and list the application. Again, doing this more than around 3 times would cause a Permgen Memory error and we'd have to go manually kill the java process anyway. That's why we're trying to kill the server itself rather than the application.

Our next idea was to restart the server every time. Our consistent problem has been that the shutdown of the server doesn't seem to kill the java process. Eventually, memory just runs out with all of those zombie processes, unless you monitor and kill them manually. The implementations I've tried are as follows:

1. Executing the bootstrap jar directly.

   <target name="tomcat.server.start" depends="tomcat.server.online" unless="tomcat.server.online">
       <java jar="${tomcat.home}/bin/bootstrap.jar" fork="true">
           <jvmarg value="-Dcatalina.home=${tomcat.home}" />
       </java>
       <waitfor maxwait="30" maxwaitunit="second" timeerty="tomcat.offline">
           <socket port="8081" server="127.0.0.1" />
       </waitfor>
       <fail if="tomcat.offline">Tomcat failed to come up.</fail>
   </target>
   

2. Executing the shutdown|startup.bat files directly

   <target name="tomcat.server.start" depends="tomcat.server.online" unless="tomcat.server.online">
       <exec executable="bash" os="Windows" failonerror="true" >
           <arg line="-i ../tools/tomcat/bin/startup.bat />
       </exec>
       <waitfor maxwait="30" maxwaitunit="second" timeerty="tomcat.offline">
           <socket port="8081" server="127.0.0.1" />
       </waitfor>
       <fail if="tomcat.offline">Tomcat failed to come up.</fail>
   </target>
   

3. Executing the shutdown|startup.sh files.

   <target name="tomcat.server.start" depends="tomcat.server.online" unless="tomcat.server.online">
       <exec executable="bash" failonerror="true" >
           <arg line="-i ../tools/tomcat/bin/startup.sh" />
       </exec>
       <waitfor maxwait="30" maxwaitunit="second" timeerty="tomcat.offline">
           <socket port="8081" server="127.0.0.1" />
       </waitfor>
       <fail if="tomcat.offline">Tomcat failed to come up.</fail>
   </target>
   

All of the above methods leave the java process alive, at least as I've implemented them.

At this point I'm at a loss as to what to do. I've heard about doing exploded development where you don't reload the container or the application but I've never found out how to do that. I don't know what the accepted best practice is here.

My problem is essentially the running of Permgen space when doing testing. If I'm asking the question wrong in the first place then feel free to correct the title. I'm open to any suggestions at all.

Thanks!

Answer 1

Use the Tomcat Ant Tasks. Catalina-ant.jar is included in most Tomcat distros. Generally, it's good to combine this with the try-catch Ant tasks.

<taskdef name="start" classname="org.apache.catalina.ant.StartTask" />
<taskdef name="stop" classname="org.apache.catalina.ant.StopTask" />
<taskdef name="try" classname="ise.antelope.tasks.TryTask"/>
<try>
    <stop url="${url}"
        username="${username}"
        password="${password}"
        path="${path}"/>
    <catch>
        <echo>${path} stop failed.</echo>
    </catch>
</try>
 <try>
    <start url="${url}"
        username="${username}"
        password="${password}"
        path="${path}"/>
    <catch>
        <echo>${path} start failed.</echo>
    </catch>
</try>

If this is a pure memory issue, you might want to tweak some memory entries in the JVM startup (I've done this often, but never written about it as clearly as this or this):

-XX:+UseConcMarkSweepGC
-XX:+CMSPermGenSweepingEnabled
-XX:+CMSClassUnloadingEnabled
-XX:MaxPermSize=256m

Answer 2

There are Tomcat Ant tasks for accomplishing these things:

http://raibledesigns.com/wiki/Wiki.jsp?page=TomcatAntTasks

Maybe they can help.

Answer 3

I would suggest running Tomcat as a service. You can then use the SSHEXEC task.

Barely edited from our build.xml:

<target name="_restartService">
 <sshexec
  host="${deploy.server.host}"
  username="${deploy.username}"
  password="${deploy.password}"
  timeout="120000"
  command="restartContainer.bat ${deploy.client}"
  trust="true" />
</target>

The restartContainer.bat file calls NET STOP %serviceName%, cleans up logs, temp files, etc., and then calls NET START %serviceName%.

It works great for us in Development, Staging, and Production. You just need an SSH server installed.