Running multiple commands simultaneously from python

Question

I want to run three commands at the same time from python. The command format is query.pl -args

Currently I am doing

os.system("query.pl -results '10000' -serverName 'server1' >> log1.txt")

os.system("query.pl -results '10000' -serverName 'server2' >> log2.txt")

os.system("query.pl -results '10000' -serverName 'server3' >> log3.txt")

I want to query all three servers at the same time but in this case, each command executes only after the last one has finished. How can I make them simultaneous? I was thinking of using '&' at the end but I want the next part of the code to be run only when all three command finish

Answer 1

You could use the subprocess module and have all three running independently: use subprocess.Popen. Take care in setting the "shell" parameter correctly.

Use the wait() or poll() method to determine when the subprocesses are finished.

Answer 2

os.system("query.pl -results '10000' -serverName 'server1' &") 
os.system("query.pl -results '10000' -serverName 'server2' &") 
os.system("query.pl -results '10000' -serverName 'server3' &")

in this case - process will be started in background

Answer 3

You can use Queue

tasks = ("query.pl -results '10000' -serverName 'server1'",\
"query.pl -results '10000' -serverName 'server2'",\
"query.pl -results '10000' -serverName 'server1'")

def worker():
    while True:
        item = q.get()
        os.system(item)

q = Queue()
for i in tasks:
     t = Thread(target=worker)
     t.setDaemon(True)
     t.start()

for item in tasks:
    q.put(item)

q.join()