Daemon dies unexpectedly

Question

I have a python script, which I daemonise using this code

        def daemonise():
            from os import fork, setsid, umask, dup2
            from sys import stdin, stdout, stderr

            if fork(): exit(0)
            umask(0) 
            setsid() 
            if fork(): exit(0)

            stdout.flush()
            stderr.flush()
            si = file('/dev/null', 'r')
            so = file('daemon-%s.out'%os.getpid(), 'a+')
            se = file('daemon-%s.err'%os.getpid(), 'a+')
            dup2(si.fileno(), stdin.fileno())
            dup2(so.fileno(), stdout.fileno())
            dup2(se.fileno(), stderr.fileno())
            print 'this file has the output from daemon%s'%os.getpid()
            print >> stderr, 'this file has the errors from daemon%s'%os.getpid()

The script is in

while True: try: funny_code(); sleep(10); except:pass;

loop. It runs fine for a few hours and then dies unexpectedly. How do I go about debugging such demons, err daemons.

[Edit]

Without starting a process like monit, is there a way to write a watchdog in python, which can watch my other daemons and restart when they go down? (Who watches the watchdog.)

Answer 1

What I've used in my clients is daemontools. It is a proven, well tested tool to run anything daemonized.

You just write your application without any daemonization, to run on foreground; Then create a daemontools service folder for it, and it will discover and automatically restart your application from now on, and every time the system restarts.

It can also handle log rotation and stuff. Saves a lot of tedious, repeated work.

Answer 2

Why are you silently swallowing all exceptions? Try to see what exceptions are being caught by this:

while True:
    try:
        funny_code()
        sleep(10)
    except BaseException, e:
        print e.__class__, e.message
        pass

Something unexpected might be happening which is causing it to fail, but you'll never know if you blindly ignore all the exceptions.

I recommend using supervisord (written in Python, very easy to use) for daemonizing and monitoring processes. Running under supervisord you would not have to use your daemonise function.

Answer 3

You really should use python-daemon for this which is a library that implements PEP 3141 for a standard daemon process library. This way you will ensure that your application does all the right things for whichever type of UNIX it is running under. No need to reinvent the wheel.