How can I manage these subprocesses efficiently?

Question

I've got a script that downloads and then converts videos. I'm trying to do all of the downloads at once(a lot of wgets) and then when they're done, convert them. Right now I have to wait while each file downloads individually and then convert when done. I want all the download requests to run concurrently.

Here's the part in my 'download' script that is holding everything up.

for id  in ids:
    subprocess.Popen(cmd, shell=True).wait()

I have my 'convert' script waiting on the download script to finish. (which is necessary, because i'm using sets of downloads to organize everything.)

I could use a queue but this I've already made this a big enough mess and I'm hoping there is a simpler solution.

Thanks

Answer 1

You could use xargs as a simple solution, if that is availabe for your platform.

From the man page:

   -P max-procs
          Run  up  to max-procs processes at a time; the default is 1.  If
          max-procs is 0, xargs will run as many processes as possible  at
          a  time.   Use the -n option with -P; otherwise chances are that
          only one exec will be done.

There are certainly better solutions, but if you need a quick one this is probably the easiest to implement.

Answer 2

you could use

process_list = []    
for id  in ids:
    process_list.append(subprocess.Popen(cmd, shell=True))
for process in process_list:
    process.wait()

as such you will wait just at the end of simultaneous jobs.