How can I setup a system to tell me if a cron job is NOT running fine?

Question

Hi,

This is more of an "general architecture" problem. If you have a cron job (or even a Windows scheduled task) running periodically, its somewhat simple to have it send you an email / text message that all is well, but how do I get informed when everything is NOT okay? Basically, if the job doesn't run at its scheduled time or Windows / linux has its own set of hangups that prevent the task from running...?

Just seeking thoughts of people who've faced this situation before and come up with interesting solutions...

Answer 1

A way I've done it in the past is to simply put at the top of each script (say, checkUsers.sh):

touch /tmp/lastrun/checkUsers.sh

then have another job that runs periodically that uses find to locate all those "marker" files in tmp/lastrun that are older than a day.

You can fiddle with the timings, having /tmp/lastrun/hour/ and tmp/lastrun/day/ to separate jobs that have different schedules.

Note that this won't catch scripts that have never run since they will never create the initial file for find-ing. To alleviate that, you can either:

• create that file manually when creating the cron job (won't handle situations where someone inadvertently deletes the marker file); or
• maintain a list of required marker files somewhere so that you can detect when they're missing as well as outdated.

And, if your cron job is not a script, put the touch directly into crontab:

0 4 * * * ( touch /tmp/lastrun/daily/checkUsers ; /usr/bin/checkUsers )

It's a lot easier to validate a simple find script than to validate every one of your cron jobs.