Python's subprocess.Popen object hangs gathering child output when child process does not exit

Question

When a process exits abnormally or not at all, I still want to be able to gather what output it may have generated up until that point.

The obvious solution to this example code is to kill the child process with an os.kill, but in my real code, the child is hung waiting for NFS and does not respond to a SIGKILL.

#!/usr/bin/python
import subprocess
import os
import time
import signal
import sys
child_script = """
#!/bin/bash
i=0
while [ 1 ]; do
    echo "output line $i"
    i=$(expr $i \+ 1)
    sleep 1
done
"""
childFile = open("/tmp/childProc.sh", 'w')
childFile.write(child_script)
childFile.close()

cmd = ["bash", "/tmp/childProc.sh"]
finish = time.time() + 3
p = subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE, stdin=subprocess.PIPE)
while p.poll() is None:
    time.sleep(0.05)
    if finish < time.time():
        print "timed out and killed child, collecting what output exists so far"
        out, err = p.communicate()
        print "got it"
        sys.exit(0)

In this case, the print statement about timing out appears and the python script never exits or progresses. Does anybody know how I can do this differently and still get output from my child processe

Answer 1

Problem is that bash doesn't answer to CTRL-C when not connected with a terminal. Switching to SIGHUP or SIGTERM seems to do the trick:

cmd = ["bash", 'childProc.sh']
p = subprocess.Popen(cmd, stdout=subprocess.PIPE, 
                          stderr=subprocess.STDOUT, 
                          close_fds=True)
time.sleep(3)
print 'killing pid', p.pid
os.kill(p.pid, signal.SIGTERM)
print "timed out and killed child, collecting what output exists so far"
out  = p.communicate()[0]
print "got it", out

Outputs:

killing pid 5844
timed out and killed child, collecting what output exists so far
got it output line 0
output line 1
output line 2

Answer 2

http://stackoverflow.com/questions/874815/how-do-i-get-real-time-information-back-from-a-subprocess-popen-in-python-2-5

Answer 3

Here's a POSIX way of doing it without the temporary file. I realize that subprocess is a little superfluous here, but since the original question used it...

import subprocess
import os
import time
import signal
import sys

pr, pw = os.pipe()
pid = os.fork () 

if pid: #parent
    os.close(pw)
    cmd = ["bash"]
    finish = time.time() + 3
    p = subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE, stdin=pr, close_fds=True)
    while p.poll() is None:
        time.sleep(0.05)
        if finish < time.time():
            os.kill(p.pid, signal.SIGTERM)
            print "timed out and killed child, collecting what output exists so far"
            out, err = p.communicate()
            print "got it: ", out
            sys.exit(0)

else: #child
    os.close(pr)
    child_script = """
    #!/bin/bash
    while [ 1 ]; do
        ((++i))
        echo "output line $i"
        sleep 1
    done
    """
    os.write(pw, child_script)

Answer 4

I had the exact same problem. I ended up fixing the issue (after scouring Google and finding many related problems) by simply setting the following parameters when calling subprocessPopen (or call):

stdout=None

and

stderr=None

There are many problems with these functions but in my specific case I believe stdout was being filled up by the process I was calling and then resulting in a blocking condition. By setting these to none (opposed to something like subprocess.PIPE) I believe this is avoided.

Hope this helps someone.