subprocess with multiple parallel jobs

Question

I'm running some subprocesses from python in parallel. I want to wait until every subprocess have finished. I'm doing a non elegant solution:

runcodes = ["script1.C", "script2.C"]
ps = []
for script in runcodes:
  args = ["root", "-l", "-q", script]
  p = subprocess.Popen(args)
  ps.append(p)
while True:
  ps_status = [p.poll() for p in ps]
  if all([x is not None for x in ps_status]):
    break

is there a class that can handle multiple subprocess? The problem is that the wait method block my program.

update: I want to show the progress during the computation: something like "4/7 subprocess finished..."

If you are curious root compile the c++ script and execute it.

Answer 1

You could do something like this:

runcodes = ["script1.C", "script2.C"]

ps = []
for script in runcodes:
    args = ["root", "-l", "-q", script]
    p = subprocess.Popen(args)
    ps.append(p)

for p in ps:
    p.wait()

The processes will run in parallel, and you'll wait for all of them at the end.

Answer 2

You could use os.wait:

test.py:

import subprocess
import os
import sys
import time
import random

# This spawns many (3) children in quick succession
# and then reports as each child finishes.
if __name__=='__main__':
    if len(sys.argv)>1:
        x=random.randint(1,10)
        print('sleeping for {x} sec'.format(x=x))
        time.sleep(x)
    else:
        pids=set()
        for script in xrange(3):            
            args=['test.py','sleep'] 
            p = subprocess.Popen(args)
            pids.add(p.pid)            
        while pids:
            pid,retval=os.wait()
            print('{p} finished'.format(p=pid))
            pids.remove(pid)

After forking all the subprocesses, os.wait() waits for any of the children processes to finish, and returns the child's PID and a return value. If you record the PIDs as you forked them, you can check them off as they come back in.

Answer 3

If your platform is not Windows, you could probably select against the stdout pipes of your subprocesses. Your app will then block until either:

• One of the registered file descriptors has an I/O event (in this case, we're interested in a hangup on the subprocess's stdout pipe)
• The poll times out

Non-fleshed-out example using epoll with Linux 2.6.xx:

import subprocess
import select

poller = select.epoll()
subprocs = {} #map stdout pipe's file descriptor to the Popen object

#spawn some processes
for i in xrange(5):
    subproc = subprocess.Popen(["mylongrunningproc"], stdout=subprocess.PIPE)
    subprocs[subproc.stdout.fileno()] = subproc
    poller.register(subproc.stdout, select.EPOLLHUP)

#loop that polls until completion
while True:
    for fd, flags in poller.poll(timeout=1): #never more than a second without a UI update
        done_proc = subprocs[fd]
        poller.unregister(fd)
        print "this proc is done! blah blah blah"
        ...  #do whatever
    #print a reassuring spinning progress widget
    ...
    #don't forget to break when all are done

Answer 4

How about

import os, subprocess
runcodes = ["script1.C", "script2.C"]
ps = {}
for script in runcodes:
    args = ["root", "-l", "-q", script]
    p = subprocess.Popen(args)
    ps[p.pid] = p
print "Waiting for %d processes..." % len(ps)
while ps:
    pid, status = os.wait()
    if pid in ps:
        del ps[pid]
        print "Waiting for %d processes..." % len(ps)