How to tell process id within Python

Question

Hey,

I am working with a cluster system over linux (www.mosix.org) that allows me to run jobs and have the system run them on different computers. Jobs are run like so:

mosrun ls &

This will naturally create the process and run it on the background, returning the process id, like so:

[1] 29199

Later it will return. I am writing a Python infrastructure that would run jobs and control them. For that I want to run jobs using the mosrun program as above, and save the process ID of the spawned process (29199 in this case). This naturally cannot be done using os.system or commands.getoutput, as the printed ID is not what the process prints to output... Any clues?

Edit:

Since the python script is only meant to initially run the script, the scripts need to run longer than the python shell. I guess it means the mosrun process cannot be the script's child process. Any suggestions?

Thanks

Answer 1

Use subprocess module. Popen instances have a pid attribute.

Answer 2

Looks like you want to ensure the child process is daemonized -- PEP 3143, which I'm pointing to, documents and points to a reference implementation for that, and points to others too.

Once your process (still running Python code) is daemonized, be it by the means offered in PEP 3143 or others, you can os.execl (or other os.exec... function) your target code -- this runs said target code in exactly the same process which we just said is daemonized, and so it keeps being daemonized, as desired.

The last step cannot use subprocess because it needs to run in the same (daemonized) process, overlaying its executable code -- exactly what os.execl and friends are for.

The first step, before daemonization, might conceivably be done via subprocess, but that's somewhat inconvenient (you need to put the daemonize-then-os.exec code in a separate .py): most commonly you'd just want to os.fork and immediately daemonize the child process.

subprocess is quite convenient as a mostly-cross-platform way to run other processes, but it can't really replace Unix's good old "fork and exec" approach for advanced uses (such as daemonization, in this case) -- which is why it's a good thing that the Python standard library also lets you do the latter via those functions in module os!-)

Answer 3

Thanks all for the help. Here's what I did in the end, and seems to work ok. The code uses python-daemon. Maybe something smarter should be done about transferring the process id from the child to the father, but that's the easier part.

import daemon
def run_in_background(command, tmp_dir="/tmp"):

    # Decide on a temp file beforehand
    warnings.filterwarnings("ignore", "tempnam is a potential security")
    tmp_filename = os.tempnam(tmp_dir)

    # Duplicate the process
    pid = os.fork()


    # If we're child, daemonize and run
    if pid == 0:
        with daemon.DaemonContext():
            child_id = os.getpid()
            file(tmp_filename,'w').write(str(child_id))
            sp = command.split(' ')
            os.execl(*([sp[0]]+sp))
    else:
        # If we're a parent, poll for the new file
        n_iter = 0
        while True:
            if os.path.exists(tmp_filename):
                child_id = int(file(tmp_filename, 'r').read().strip())
                break

            if n_iter == 100:
                raise Exception("Cannot read process id from temp file %s" % tmp_filename)
            n_iter += 1

            time.sleep(0.1)

        return child_id