Non-blocking read on a stream in python.

Question

Hi,

I'm using the subprocess module to start a subprocess and connect to it's output stream (stdout). I want to be able to execute non-blocking reads on its stdout. Is there a way to make .readline non-bloking or to check if there is data on the stream before I invoke .readline? I'd like this to be portable or at least work under Windows and Linux.

here is how I do it for now (It's blocking on the .readline if no data is avaible):

p = subprocess.Popen('myprogram.exe', stdout = subprocess.PIPE)
str = p.stdout.readline()

Thanks for your help.

Answer 1

One solution is to make another process to perform your read of the process, or make a thread of the process with a timeout.

Here's the threaded version of a timeout function:

http://code.activestate.com/recipes/473878/

However, do you need to read the stdout as it's coming in? Another solution may be to dump the output to a file and wait for the process to finish using p.wait().

f = open('myprogram_output.txt','w')
p = subprocess.Popen('myprogram.exe', stdout=f)
p.wait()
f.close()


str = open('myprogram_output.txt','r').read()

Answer 2

The select module helps you determine where the next useful input is.

However, you're almost always happier with separate threads. One does a blocking read the stdin, another does wherever it is you don't want blocked.

Answer 3

Try the asyncproc module. For example:

import os
from asynproc import Process
myProc = Process("myprogram.app")

while True:
    # check to see if process has ended
    poll = myProc.wait(os.WNOHANG)
    if poll != None:
        break
    # print any new output
    out = myProc.read()
    if out != "":
        print out

The module takes care of all the threading as suggested by S.Lott.

Answer 4

Looks like you want p.communicate() (see the docs)

Answer 5

I have often had a similar problem; Python programs I write frequently need to have the ability to execute some primary functionality while simultaneously accepting user input from the command line (stdin). Simply putting the user input handling functionality in another thread doesn't solve the problem because readline() blocks and has no timeout. If the primary functionality is complete and there is no longer any need to wait for further user input I typically want my program to exit, but it can't because readline() is still blocking in the other thread waiting for a line. A solution I have found to this problem is to make stdin a non-blocking file using the fcntl module:

import fcntl

# make stdin a non-blocking file
fd = sys.stdin.fileno()
fl = fcntl.fcntl(fd, fcntl.F_GETFL)
fcntl.fcntl(fd, fcntl.F_SETFL, fl | os.O_NONBLOCK)

# user input handling thread
while mainThreadIsRunning:
      try: input = sys.stdin.readline()
      except: continue
      handleInput(input)

In my opinion this is a bit cleaner than using the select or signal modules to solve this problem but then again it only works on UNIX...
Cheers,
Jesse

Answer 6

This allows you to pass a timeout to read() http://www.pixelbeat.org/libs/subProcess.py

Answer 7

I have port http://code.google.com/p/subprocdev/ to python 2.5, and it works in Windows

Answer 8

Jesse's answer is not correct. According to Guido, readline doesn't work correctly with non-blocking mode, and it won't before Python 3000.

http://bugs.python.org/issue1175#msg56041

If you want to use fcntl to set the file to non-blocking mode, you have to use the lower-level os.read() and separate out the lines yourself. Mixing fcntl with high-level calls that perform line buffering is asking for trouble.