Weblogic server 10.0 - Managed Server shutting down

Question

We have a weblogic server 10.0 instance which has a cluster with one managed server. Every Monday at 5am (or few seconds after), it shuts down on its own. The logs do not show any errors except the following message:

JVM called WLS shutdown hook. The server will force shutdown now.

JVM has a -Xnohup option and using JRockit. There is no cron job on the server. I am not sure how to debug this one. Admin Server keeps running with no issues and I am able to start the managed server back up with no problems. Any help is greatly appreciated.

Update:
I tried using -Djrockit.waitonerror, but JVM does not pause. I also see the following "panic" message in the log either just before or few minutes before. CharScanner; panic: ClassNotFoundException: org.antlr.stringtemplate.language.ChunkToken. Looks like this during some kind of Diagnostics run:

<Diagnostics> <BEA-320140> <Scheduling data retirement tasks as per configuration.>

Thanks.

Answer 1

Since you are using -Xnohup option, the JVM is very likely not receiving an OS signal. Still, something is triggering the shutdown (and the fact that it happens at very regular intervals suggests an automated process).

Try to apply the second suggestion of this blog post (I think that this come from the support actually):

If the issue recurs again even after trying the above option then can you please apply the following JAVA_OPTION:

• Depending on the JVM version, it may be possible to get a thread dump before the process exits

  • HotSpot supports the command-line option -XX:+ShowMessageBoxOnError

  • The corresponding JRockit option is -Djrockit.waitonerror

  • While the JVM goes down, it may prompt the user: "Do you want to debug the problem?"

  • This pauses the JVM, thereby creating an opportunity to generate a thread dump (a stack trace of every thread in the JVM), attach a debugger, or perform some other debugging activity.

To generate a thread dump, I think that you'll have to remove the -Xnohup option (this shouldn't be a problem since it doesn't help). This would really help the analysis.

Also open a case, the support might have more input on this (you're not the only one facing this problem, see the mentioned threads below).

---

Just in case, have a look at WebLogic Server shutting down Automatically (any watchdog process OOMKiller running?) and JVM called WLS shutdown hook. The server will force shutdown now.

Answer 2

Conflict with weblogic's antlr version and the one used in the deployed app (war). Included that war in an ear file and put a weblogic-application.xml file in META-INF with the following:

<weblogic-application xmlns="http://www.bea.com/ns/weblogic/90" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"&gt;      
<prefer-application-packages>
    <package-name>antlr.*</package-name>
</prefer-application-packages>