subprocess with timeout

Question

In Python, I have a program snippet similar to the following that has the effect of running a custom command and returning the stdout data (or raise exception when exit code is non-zero):

proc = subprocess.Popen(
    cmd,
    # keep stderr separate; or merge it with stdout (default).
    stderr=(subprocess.PIPE if ignore_stderr else subprocess.STDOUT),
    stdout=subprocess.PIPE,
    shell=True)

And then I use communicate (not wait which could deadlock) to wait for the complete stdout data:

stdoutdata, stderrdata = proc.communicate()

My question is - how do I set a timeout for any command? For example, I don't want the program to wait indefinitely because a particular programs takes more than, say, 5 minutes to run.

Simpler, unsophisticated solutions would be nice.

Answer 1

If you're on Unix,

import signal
  ...
class Alarm(Exception):
    pass

def alarm_handler(signum, frame):
    raise Alarm

signal.signal(signal.SIGALARM, alarm_handler)
signal.alarm(5*60)  # 5 minutes
try:
    stdoutdata, stderrdata = proc.communicate()
    signal.alarm(0)  # reset the alarm
except Alarm:
    print "Oops, taking too long!"
    # whatever else

Answer 2

This is the best I could come up with (extracted from my private program):

        # poll for terminated status till timeout is reached
        t_beginning = time.time()
        seconds_passed = 0
        while True:
            if p.poll() is not None:
                break
            seconds_passed = time.time() - t_beginning
            if timeout and seconds_passed > timeout:
                p.terminate()
                raise TimeoutError(cmd, timeout)
            time.sleep(0.1)

(inspired by some other SO comment elsewhere)

Answer 3

I've used killableprocess successfully on Windows, Linux and Mac. If you are using Cygwin Python, you'll need OSAF's version of killableprocess because otherwise native Windows processes won't get killed.

Answer 4

Here is Alex Martelli's solution as a module with proper process killing. The other approaches do not work because they do not use proc.communicate(). So if you have a process that produces lots of output, it will fill its output buffer and then block until you read something from it.

from os import kill
from signal import alarm, signal, SIGALRM, SIGKILL
from subprocess import PIPE, Popen

def run(args, cwd = None, shell = False, kill_tree = True, timeout = -1):
    '''
    Run a command with a timeout after which it will be forcibly
    killed.
    '''
    class Alarm(Exception):
        pass
    def alarm_handler(signum, frame):
        raise Alarm
    p = Popen(args, shell = shell, cwd = cwd, stdout = PIPE, stderr = PIPE)
    if timeout != -1:
        signal(SIGALRM, alarm_handler)
        alarm(timeout)
    try:
        stdout, stderr = p.communicate()
        if timeout != -1:
            alarm(0)
    except Alarm:
        pids = [p.pid]
        if kill_tree:
            pids.extend(get_process_children(p.pid))
        for pid in pids:
            kill(pid, SIGKILL)
        return -9, '', ''
    return p.returncode, stdout, stderr

def get_process_children(pid):
    p = Popen('ps --no-headers -o pid --ppid %d' % pid, shell = True,
              stdout = PIPE, stderr = PIPE)
    stdout, stderr = p.communicate()
    return [int(p) for p in stdout.split()]

if __name__ == '__main__':
    print run('find /', shell = True, timeout = 3)
    print run('find', shell = True)