Sometimes when I redeploy war too many times, jboss gives java.lang.OutOfMemoryError: PermGen space error, is it possible to monitor jboss with other Java program that is not run inside jboss, to make sure it has not run ot of memory and if it is, then automatically restart jboss?
I would expect that you can monitor the memory consumption via JMX and the MemoryMXBean. You can do this interactively via JConsole, or code up a simple monitor to do this automatically.
Here's some details on how to do this in-process, but you can do this remotely as well. See the JMX docs for more info.
Alternatively, you can run a process under the JavaServiceWrapper, and get it to shutdown/restart a process depending on messages coming out from stdout/err. That may be a simple way to perform your restart automatically. However, I'd prefer using the JMX solution in the long term so you can get advance warning of issues (and perhaps tie them to their underlying cause).
+1 for using a standalone application. If HypericHQ it too much of a beast for you, try something simpler like Service Hawk. You can use it to restart the JBoss service(s) nightly to keep it running well.