How do you go about setting up monitoring for a non-web frontend process?

Question

I have a worker process that is running in a server with no web frontend. what is the best way to set up monitoring fot it? It recently died for 3 days, and i did not know about it

Answer 1

There are several ways to do this. One simple option is to run a cron job that checks timestamps on the process's logs (and, of course, make sure the process logs something routinely).

Answer 2

Roll your own reincarnation job. Have your background process get its PID, then write it to a specific pre-determined location when it starts. Have another process (or perhaps cron) read that PID, then check the symbolic link proc/{pid}/exe. If that link does not exist or is not your process, it needs to be re-started.

With PHP, you can use posix_getpid() to obtain the PID. Then fopen() / fwrite() to write it to a file. use readlink() to read the symbolic link (take care to note FALSE as a return value).

Here's a simple bash-ified example of how the symlink works:

tpost@tpost-desktop:~$ echo $$
13737
tpost@tpost-desktop:~$ readlink /proc/13737/exe
/bin/bash

So, once you know the PID that the process started with, you can check to see if its still alive, and that Linux has not recycled the PID (you only see PID recycling on systems that have been running for a very long time, but it does happen).

This is a very, very cheap operation, feel free to let it do its work every minute, 30 seconds or even shorter intervals.