Python-daemon doesn't kill its kids

Question

When using python-daemon, I'm creating subprocesses likeso:

import multiprocessing

class Worker(multiprocessing.Process):
   def __init__(self, queue):
      self.queue = queue # we wait for things from this in Worker.run()

   ...

q = multiprocessing.Queue()

with daemon.DaemonContext():
    for i in xrange(3):
       Worker(q)

    while True: # let the Workers do their thing
       q.put(_something_we_wait_for())

When I kill the parent daemonic process (i.e. not a Worker) with a Ctrl-C or SIGTERM, etc., the children don't die. How does one kill the kids?

My first thought is to use atexit to kill all the workers, likeso:

 with daemon.DaemonContext():
    workers = list()
    for i in xrange(3):
       workers.append(Worker(q))

    @atexit.register
    def kill_the_children():
        for w in workers:
            w.terminate()

    while True: # let the Workers do their thing
       q.put(_something_we_wait_for())

However, the children of daemons are tricky things to handle, and I'd be obliged for thoughts and input on how this ought to be done.

Thank you.

Answer 1

Atexit won't do the trick -- it only gets run on successful non-signal termination -- see the note near the top of the docs. You need to set up signal handling via one of two means.

The easier-sounding option: set the daemon flag on your worker processes, per http://docs.python.org/library/multiprocessing.html#process-and-exceptions

Somewhat harder-sounding option: PEP-3143 seems to imply there is a built-in way to hook program cleanup needs in python-daemon.

Answer 2

Your options are a bit limited. If doing self.daemon = True in the constructor for the Worker class does not solve your problem and trying to catch signals in the Parent (ie, SIGTERM, SIGINT) doesn't work, you may have to try the opposite solution - instead of having the parent kill the children, you can have the children commit suicide when the parent dies.

The first step is to give the constructor to Worker the PID of the parent process (you can do this with os.getpid()). Then, instead of just doing self.queue.get() in the worker loop, do something like this:

waiting = True
while waiting:
    # see if Parent is at home
    if os.getppid() != self.parentPID:
        # woe is me! My Parent has died!
        sys.exit() # or whatever you want to do to quit the Worker process
    try:
        # I picked the timeout randomly; use what works
        data = self.queue.get(block=False, timeout=0.1)
        waiting = False
    except queue.Queue.Empty:
        continue # try again
# now do stuff with data

The solution above checks to see if the parent PID is different than what it originally was (that is, if the child process was adopted by init or lauchd because the parent died) - see reference. However, if that doesn't work for some reason you can replace it with the following function (adapted from here):

def parentIsAlive(self):
    try:
        # try to call Parent
        os.kill(self.parentPID, 0)
    except OSError:
        # *beeep* oh no! The phone's disconnected!
        return False
    else:
        # *ring* Hi mom!
        return True

Now, when the Parent dies (for whatever reason), the child Workers will spontaneously drop like flies - just as you wanted, you daemon! :-D

Answer 3

can't you just store the parent pid when the child is first created (let's say in self.myppid) and when self.myppid is diferent from getppid() means that the parent died.

you could also use signals to avoid the need to keep checking if the parent has changed. I don't know the python specifics but something along like what is described here (at the bottom of the page) might work.