Python, Popen and select - waiting for a process to terminate or a timeout

Question

I run a subprocess using:

  p = subprocess.Popen("subprocess", 
                       stdout=subprocess.PIPE, 
                       stderr=subprocess.PIPE, 
                       stdid=subprocess.PIPE)

This subprocess could either exit immediately with an error on stderr, or keep running. I want to detect either of these conditions - the latter by waiting for several seconds.

I tried this:

  SECONDS_TO_WAIT = 10
  select.select([], 
                [p.stdout, p.stderr], 
                [p.stdout, p.stderr],
                SECONDS_TO_WAIT)

but it just returns:

  ([],[],[])

on either condition. What can I do?

Answer 1

Have you tried using the Popen.Poll() method. You could just do this:

p = subprocess.Popen("subprocess", 
                   stdout=subprocess.PIPE, 
                   stderr=subprocess.PIPE, 
                   stdid=subprocess.PIPE)

time.sleep(SECONDS_TO_WAIT)
retcode = p.Poll()
if retcode:
   # process has terminated

This will cause you to always wait 10 seconds, but if the failure case is rare this would be amortized over all the success cases.

---

Edit:

How about:

t_nought = time.time()
seconds_passed = 0

while(not p.Poll() and seconds_passed < 10):
    seconds_passed = time.time() - t_nought

if seconds_passed >= 10:
   #success

This has the ugliness of being a busy wait, but I think it accomplishes what you want.

Additionally looking at the select call documentation again I think you may want to change it as follows:

SECONDS_TO_WAIT = 10
  select.select([p.stderr], 
                [], 
                [p.stdout, p.stderr],
                SECONDS_TO_WAIT)

Since you would typically want to read from stderr, you want to know when it has something available to read (ie the failure case).

I hope this helps.

Answer 2

Using select and sleeping doesn't really make much sense. select (or any kernel polling mechanism) is inherently useful for asynchronous programming, but your example is synchronous. So either rewrite your code to use the normal blocking fashion or consider using Twisted:

from twisted.internet.utils import getProcessOutputAndValue
from twisted.internet import reactor    

def stop(r):
    reactor.stop()
def eb(reason):
    reason.printTraceback()
def cb(result):
    stdout, stderr, exitcode = result
    # do something
getProcessOutputAndValue('/bin/someproc', []
    ).addCallback(cb).addErrback(eb).addBoth(stop)
reactor.run()

Incidentally, there is a safer way of doing this with Twisted by writing your own ProcessProtocol:

http://twistedmatrix.com/projects/core/documentation/howto/process.html

Answer 3

If, as you said in the comments above, you're just tweaking the output each time and re-running the command, would something like the following work?

from threading import Timer
import subprocess

WAIT_TIME = 10.0

def check_cmd(cmd):
    p = subprocess.Popen(cmd,
     stdout=subprocess.PIPE, 
            stderr=subprocess.PIPE)
    def _check():
     if p.poll()!=0:
      print cmd+" did not quit within the given time period."

    # check whether the given process has exited WAIT_TIME
    # seconds from now
    Timer(WAIT_TIME, _check).start()

check_cmd('echo')
check_cmd('python')

The code above, when run, outputs:

python did not quit within the given time period.

The only downside of the above code that I can think of is the potentially overlapping processes as you keep running check_cmd.