What is the best way to run multiple subprocesses via fork()?

Question

A python script need to spawn multiple sub-processes via fork(). All of those child processes should run simultaneously and the parent process should be waiting for all of them to finish. Having an ability to set some timeout on a "slow" child would be nice. The parent process goes on processing the rest of the script after all kids are collected.

What is the best way to work it out? Thanks.

Answer 1

Have you looked at the pyprocessing module?

Answer 2

The traditional, UNIX-y way to communicate with sub-processes is to open pipes to their standard input/output, and use the select() system call to multiplex the communications in the parent process (available in Python via... the select module).

If you need to kill a slow-running child process, you can just save its process ID (returned by the os.fork() call) and then use os.kill() to kill it when not needed anymore. Of course, it would probably be cleaner to be able to communicate with the child process explicitly and tell it to shut itself down.

Answer 3

Simple example:

import os
chidren = []
for job in jobs:
    child = os.fork()
    if child:
        children.append(child)
    else:
        pass  # really should exec the job
for child in children:
    os.waitpid(child, 0)

Timing out a slow child is a little more work; you can use wait instead of waitpid, and cull the returned values from the list of children, instead of waiting on each one in turn (as here). If you set up an alarm with a SIGALRM handler, you can terminate the waiting after a specified delay. This is all standard UNIX stuff, not Python-specific...

Answer 4

Ephemient: each child in your code will stay in the for loop after his job ends. He will fork again and again. Moreover, the children that start when children[] is not empty will try to wait for some of their brothers at the end of the loop. Eventually someone will crash. This is a workaround:

import os, time

def doTheJob(job):
    for i in xrange(10):
     print job, i
     time.sleep(0.01*ord(os.urandom(1)))
     # random.random() would be the same for each process

jobs = ["A", "B", "C", "D", "E", "F", "G", "H", "I", "J"]
imTheFather = True
children = []

for job in jobs:
    child = os.fork()
    if child:
     children.append(child)
    else:
     imTheFather = False
     doTheJob(job)
     break

# in the meanwhile 
# ps aux|grep python|grep -v grep|wc -l == 11 == 10 children + the father

if imTheFather:
    for child in children:
     os.waitpid(child, 0)