Trying to call readline() on a file object in python but it's pausing

Question

I'm using the readline() function to read data from a file object obtained through the subprocess module: proc = subprocess.Popen(cmd, bufsize=0, stdout=subprocess.PIPE). This allows me to use proc.stdout as a file-like object with proc.stdout.readline(). My issue is that this pauses waiting for input and I'd like it to time out and move on if there isn't input there when I make the readline call. I'm running Python 2.4, how can I get the readline method to stop pausing? Thanks.

Answer 1

On a posix-y platform (basically any popular platform except Windows), the select module offers the right tools for this purpose. Unfortunately, on Windows, select only works on sockets (not on pipes, which is what subprocess.Popen will be using), so the situation is not quite as clear there. Do you need to run on Windows...?

If not, just use the p.stdout.fileno() of your subprocess object p in a select.select call with a short timeout -- it's really easy!

Edit: here's a simple example (assuming the needed imports of course):

>>> def f():                                                                    
...   p = subprocess.Popen("sleep 10; echo ciao", shell=True, stdout=subprocess.PIPE)
...   while True:                                                               
...     r, w, x = select.select([p.stdout.fileno()],[],[],1.0)
...     if r: return p.stdout.read()
...     print 'not ready yet'
... 
>>> f()
not ready yet
not ready yet
not ready yet
not ready yet
not ready yet
not ready yet
not ready yet
not ready yet
not ready yet
not ready yet
'ciao\n'
>>> 

Note there is no way to "wait for a complete line": this waits for "any output at all" (then blocks until all the output is ready). To read just what's available, use fcntl to set os.O_NODELAY on the file descriptor (what fileno() returns) before you start looping.