Spawning and waiting for child processes in Python

Question

The relevant part of the code looks like this:

pids = [] 
for size in SIZES:
    pids.append(os.spawnv(os.P_NOWAIT, RESIZECMD, [RESIZECMD, lotsOfOptions]))

# Wait for all spawned imagemagick processes to finish
while pids:
    (pid, status) = os.waitpid(0, 0)
    if pid:
        pids.remove(pid)

What this should be doing is spawning all of the processes off, then waiting for each process to finish before continuing. What it does is work for the most part but sometimes crash on the next section (when it expects all of these processes to be finished).

Is there something wrong with this? Is there a better way of doing it?

The environment it has to work on is CentOS with Python 2.4, but I'm testing on Cygwin with Python 2.5, so it could be that it fails on my machine but will work on the Linux one (the Linux machine is very slow and this error is rare, so I haven't been able to get it on there).

Answer 1

I would recommend you install python-subprocess32 -- a robust backport of Python 3's version of the subprocess standard library module, suitable for Python 2.4 to 2.7, and by far the best way to run subprocesses in Python 2. Then, in the loop you'll do

pids.append(subprocess.Popen([RESIZECMD, lot, of, options])

and the following loop will just be a simple

for pid in pids:
    pid.wait()

Answer 2

The recommended way to start subprocess is to use the subprocess module.

pipe = Popen(["program", "arg1", "arg2"])
pipe.wait()