spawning process from python

Question

im spawning a script that runs for a long time from a web app like this:

os.spawnle(os.P_NOWAIT, "../bin/producenotify.py", "producenotify.py", "xx",os.environ)

the script is spawned successfully and it runs, but till it gets over i am not able to free the port that is used by the web app, or in other words i am not able to restart the web app. how do i spawn off a process and make it completely independent of the web app?

this is on linux os.

Answer 1

As @mark clarified it's a Linux system, the script could easily make itself fully independent, i.e., a daemon, by following this recipe. (You could also do it in the parent after an os.fork and only then os.exec... the child process).

Edit: to clarify some details wrt @mark's comment on my answer: super-user privileges are not needed to "daemonize" a process as per the cookbook recipes, nor is there any need to change the current working directory (though the code in the recipe does do that and more, that's not the crucial part -- rather it's the proper logic sequence of fork, _exit and setsid calls). The various os.exec... variants that do not end in e use the parent process's environment, so that part is easy too -- see Python online docs.

To address suggestions made in others' comments and answers: I believe subprocess and multiprocessing per se don't daemonize the child process, which seems to be what @mark needs; the script could do it for itself, but since some code has to be doing forks and setsid, it seems neater to me to keep all of the spawning on that low-level plane rather than mix some high-level and some low-level code in the course of the operation.

Here's a vastly reduced and simplified version of the recipe at the above URL, tailored to be called in the parent to spawn a daemon child -- this way, the code can be used to execute non-Python executables just as well. As given, the code should meet the needs @mark explained, of course it can be tailored in many ways -- I strongly recommend reading the original recipe and its comments and discussions, as well as the books it recommends, for more information.

import os
import sys

def spawnDaemon(path_to_executable, *args)
    """Spawn a completely detached subprocess (i.e., a daemon).

    E.g. for mark:
    spawnDaemon("../bin/producenotify.py", "producenotify.py", "xx")
    """
    # fork the first time (to make a non-session-leader child process)
    try:
        pid = os.fork()
    except OSError, e:
        raise RuntimeError("1st fork failed: %s [%d]" % (e.strerror, e.errno))
    if pid != 0:
        # parent (calling) process is all done
        return

    # detach from controlling terminal (to make child a session-leader)
    os.setsid()
    try:
        pid = os.fork()
    except OSError, e:
        raise RuntimeError("2nd fork failed: %s [%d]" % (e.strerror, e.errno))
        raise Exception, "%s [%d]" % (e.strerror, e.errno)
    if pid != 0:
        # child process is all done
        os._exit(0)

    # grandchild process now non-session-leader, detached from parent
    # grandchild process must now close all open files
    try:
        maxfd = os.sysconf("SC_OPEN_MAX")
    except (AttributeError, ValueError):
        maxfd = 1024

    for fd in range(maxfd):
        try:
           os.close(fd)
        except OSError: # ERROR, fd wasn't open to begin with (ignored)
           pass

    # redirect stdin, stdout and stderr to /dev/null
    os.open(REDIRECT_TO, os.O_RDWR) # standard input (0)
    os.dup2(0, 1)
    os.dup2(0, 2)

    # and finally let's execute the executable for the daemon!
    try:
      os.execv(path_to_executable, args)
    except Exception, e:
      # oops, we're cut off from the world, let's just give up
      os._exit(255)

Answer 2

You can use the multiprocessing library to spawn processes. A basic example is shown here:

from multiprocessing import Process

def f(name):
    print 'hello', name

if __name__ == '__main__':
    p = Process(target=f, args=('bob',))
    p.start()
    p.join()

Answer 3

(Sorry, I can't yet comment on Alex' answer directly.)

In my application, the first child turns into a zombie with Alex' code. Before the "return", I inserted "os.waitpid(pid, 0)", which worked for me. But I'd appreciate if someone could confirm this as I have only limited knowledge about Unix processes. In particular, I haven't understood the "option" argument of waitpid.