How to make sure an application keeps running on Linux

Question

I'm trying to ensure a script remains running on a development server. It collates stats and provides a web service so it's supposed to persist, yet a few times a day, it dies off for unknown reasons. When we notice we just launch it again, but it's a pain in the rear and some users don't have permission (or the knowhow) to launch it up.

The programmer in me wants to spend a few hours getting to the bottom of the problem but the busy person in me thinks there must be an easy way to detect if an app is not running, and launch it again.

I know I could cron-script ps through grep:

ps -A | grep appname

But again, that's another hour of my life wasted on doing something that must already exist... Is there not a pre-made app that I can pass an executable (optionally with arguments) and that will keep a process running indefinitely?

In case it makes any difference, it's Ubuntu.

Answer 1

Monit is perfect for this :)

You can write simple config files which tell monit to watch e.g. a TCP port, a PID file etc

monit will run a command you specify when the process it is monitoring is unavailable/using too much memory/is pegging the CPU for too long/etc. It will also pop out an email alert telling you what happened and whether it could do anything about it.

We use it to keep a load of our websites running while giving us early warning when something's going wrong.

-- Your faithful employee, Monit

Answer 2

It's a job for a DMD (daemon monitoring daemon). there are a few around; but I usually just write a script that checks if the daemon is running, and run if not, and put it in cron to run every minute.

Answer 3

first of all, how do you start this app? Does it fork itself to the background? Is it started with nohup .. & etc? If it's the latter, check why it died in nohup.out, if it's the first, build logging.

As for your main question: you could cron it, or run another process on the background (not the best choice) and use pidof in a bashscript, easy enough:

if [ `pidof -s app` -eq 0 ]; then
    nohup app &
fi

Answer 4

Put your run in a loop- so when it exits, it runs again... while(true){ run my app.. }

Answer 5

You could make it a service launched from inittab (although some Linuxes have moved on to something newer in /etc/event.d). These built in systems make sure your service keeps running without writing your own scripts or installing something new.

Answer 6

Check out 'nanny' referenced in Chapter 9 (p197 or thereabouts) of "Unix Hater's Handbook" (one of several sources for the book in PDF).

Answer 7

Since you're using ubuntu, you may be interested in Upstart, which has replaced the traditional sysV init. One key feature is that it can restart a service if it dies unexpectedly. Fedora has moved to upstart, and debian is in experimental, so it may be work looking into.

This may be overkill for this situation though, as a cron script will take 2 minutes to implement.

#!/bin/bash
if [[ ! `pidof -s yourapp` ]]; then
    invoke-rc.d yourapp start
fi

Answer 8

Also check out daemontools

http://cr.yp.to/daemontools.html

Answer 9

I have used from cron "killall -0 programname || /etc/init.d/programname start". kill will error if the process doesn't exist. If it does exist, it'll deliver a null signal to the process (which the kernel will ignore and not bother passing on.)

This idiom is simple to remember (IMHO). Generally I use this while I'm still trying to discover why the service itself is failing. IMHO a program shouldn't just disappear unexpectedly :)

Answer 10

I have used a simple script with cron to make sure that the program is running. If it is not, then it will start it up. This may not be the perfect solution you are looking for, but it is simple and works rather well.

#!/bin/bash
#make-run.sh
#make sure a process is always running.

export DISPLAY=:0 #needed if you are running a simple gui app.

process=YourProcessName
makerun="/usr/bin/program"

if ps ax | grep -v grep | grep $process > /dev/null
        then
                exit
        else
        $makerun &
        fi
exit

Then add a cron job every minute, or every 5 minutes.

Answer 11

thanks lordbinky that worked great for me!