What is the best way to monitor (java) process deaths on a Windows box?

Question

We have a curious problem with our java processes dying.

The application doesn't stacktrace, or write anything to the logs, the process just randomly dies. It's a heavily used application, but the problem only appears about once a month.

We're currently looking into using Process Monitor but any other suggestions would be welcome.

Edit:

It's a distributed Java application, running on Weblogic with an in-house web framework (Yes, this is a terrible idea, but it's been running for eight years), connecting to Oracle.

-

Out of Memory? Our logs would catch java.lang.OutOfMemoryException, according to Brian Agnew.

Write crashes to a log? I don't think Java ever gets the chance, the death is happening at a process level, rather than Java exiting.

Answer 1

You can use a Linux NAGIOS Server to monitor the health of your Windows machines and services! Have a look at: nagios-monitoring-windows.

If you have such problems with your java app! You should test it and debug it! Applications shouldn't die without a trace! Look for logfiles! From which vendor is the app? Or is it self written? Try to enforce another Log4J/Logger/Debug Level. Monitor your System with cacti etc. to reduce the possibilities for such a crash. Talk to the software vendor.

Is enogh memory available? Maybe the app runs out of memory? Is it a standalone java process or a java process from a tomcat/jboss server?

Have you written down the crash times to a log? Appear they in different time-slices? Or appear they nearly time-circular?

Answer 2

Can you wrap it in some shell script that captures the log files (stdout/stderr) and the exit code (which should give some indication as to how it died) ? On JVM exit you can also capture machine level stats using WMI

IF the VM itself is crashing it'll leave behind an hs_err_pid... file that contains stacktraces, machine-level debug info. You can then use that to diagnose the VM issue. See this blog entry for further information.

If the problem is related to the app's behaviour, it may be worth looking at JConsole, although from your description of the issue, this sounds much more like a low level VM issue.

(I assume you're on the latest VM for your Java version number etc.)

Answer 3

VisualVM is a new tool which makes monitoring Java applications easier:

https://visualvm.dev.java.net/description.html

"VisualVM is a tool that provides detailed information about Java applications while they are running. It provides an intuitive graphical user interface that allows you to easily see information about multiple Java applications."