How to execute a Java program 24 x 7 on linux

Question

Hello
I have a developed two small Java applications - a vanilla Java app and a Java Web application (i.e. Spring MVC, Servlets, JSP, etc.).
The vanilla application consists of serveral threads which read data continously at varying rates (from once a second to twice a minute) from several websites, process the data and write it to a database.
The Web Application reads the data from the database and presents it using JSPs, etc.

I'd now like to deploy the applications to a Linux machine and have them run 24 x 7.
If the applications crash I would like them to be restarted.
What's the best way of doing this?
Thanks

Answer 1

Your web container will run 24x7 by default. If your deployed application throws an exception, it's captured by the container. I wouldn't normally expect this process to not run. Perhaps if threads run away, then it may become unresponsive, so it's worth monitoring (perhaps by a separate process querying it via HTTP?).

Does your vanilla application need to run at regular intervals ? If so, then cron is a good bet. It'll invoke a new instance every 'n' minutes (or however you configure it). If your instance suffers a problem, then it'll simply bail out and a new instance will be launched at the next configured interval. Again, you should probably monitor this (capture log files?) in case some problem determines that it'll never succeed completely.

Answer 2

I have a crontab job running every 15 minutes to see if the script is still running. If not, it restarts the service. The script itself is a piece of Perl code:

#!/usr/bin/perl                                                                                      

use diagnostics;
use strict;

my %is_set;
for (@ARGV) {
  $is_set{$_} = 1;
}

my $verbose = -1;
if ($is_set{"--verbose"}) {
  $verbose = 1;
}

my @components = ("cdk", "qsar", "rdf");
foreach my $comp (@components) {
  print "Checking component $comp\n" if ($verbose == 1);
  my $bla = `ps aux | grep component | grep $comp-xws | grep -v "ps aux" | wc -l`;
  $bla =~ s/\n|\r//g;
  if ($bla eq "1") {
    print " ... running\n" if ($verbose == 1);
  } else {
    print " ... restarting component $comp\n" if ($verbose == 1);
    system "cd /home/egonw/runtime/$comp; sh runCDKcomponent.sh &";
  }
}

Answer 3

with Ubuntus upstart you can respawn processes automatically. A little bit more low-level is to put the respawn directly in /etc/inittab. Both work well, but upstart is more manageable (more tools), but requires a newer system (ubuntu, fedora, and debian is switching soon).

For inittab you need to add a line like this to /etc/inittab (from the inittab manpage):

12:2345:respawn:/path/to/myapp flags

For upstart you do something similar (this is a standard script in ubuntu 9.10):

user@host:/etc/init$ cat tty1.conf

# tty1 - getty
#
# This service maintains a getty on tty1 from the point the system is
# started until it is shut down again.
start on stopped rc RUNLEVEL=[2345]
stop on runlevel [!2345]

respawn
exec /sbin/getty -8 38400 tty1

Answer 4

Check out the ServletContextListener, this allows you to embed your java application inside your web application (by creating a background thread). Then you can have it all running inside the web container.

Consider investigating and using a web container supported by the operating system vendor so all the scripts to bring it up and down (including in case of problems) is written and maintained by somebody else but you.

E.g. Ubuntu has a Tomcat as a package

Answer 5

First, when a problem occur, it is in general a good idea to have a human look at it to find the root cause as restarting a service without any action will in many cases not magically solve the issue. The common way to handle this situation is to use a monitoring solution offering some kind of alerting (by email, sms, etc) to let a human know that something is wrong and needs a human action. For example, have a look at HypericHQ, OpenNMS, Zenoss, Nagios, etc.

Second, if you want to offer some kind of highly available service, running multiple instances of the service (this is often referred to as clustering) would be a good idea. When doing so, if one instance goes down, the service won't be totally interrupted, obviously. Note that when using a cluster, if one node goes down because of too heavy load, it's very unlikely that the remaining part of the cluster will be able to handle the load so clustering isn't an absolute guarantee in all situations. Implementing this (at least for the web application) depends on the application server or servlet engine you are using.

But actually, if you are looking for something simple and pretty straight forward, I'd warmly suggest to check monit which is really a better alternative to a custom cron job (don't reinvent the wheel, monit is precisely doing what you want in a smart way). See this article for an introduction:

monit is a utility for managing and monitoring processes, files, directories and devices on a Unix system. Monit conducts automatic maintenance and repair and can execute meaningful causal actions in error situations. For example, monit can start a process if it does not run, restart a process if it does not respond and stop a process if it uses to much resources. You may use monit to monitor files, directories and devices for changes, such as timestamps changes, checksum changes or size changes.

Answer 6

Java Service Wrapper may help with keeping the Java program up 24x7 (or very close).

Several years ago I worked on a project using Java 1.2 and our goal was to run 24x7. We never made it. The longest we managed to keep Java running was about 2-3 weeks. Twice it crashed after about 15 days. The first time we just restarted it, the second time a colleague did some research and found that the crash was due to an int variable overflowing in the Calendar class: the JdbcDriver had called new Date(year, month, day, hour minute, second) more than about 300 million times and each call had incremented the int 6 times. I think this particular bug may be fixed but you may find there are others that you encounter as you try to keep the JVM running for a long time.

So you may need to design your application to be restarted occasionally to avoid this kind of thing.