Python - simple reading lines from a pipe

Question

I'm trying to read lines from a pipe and process them, but I'm doing something silly and I can't figure out what. The producer is going to keep producing lines indefinitely, like this:

producer.py

import time

while True:
    print 'Data'
    time.sleep(1)

The consumer just needs to check for lines periodically:

consumer.py

import sys, time
while True:
    line = sys.stdin.readline()
    if line:
        print 'Got data:', line
    else:
        time.sleep(1)

When I run this in the Windows shell as python producer.py | python consumer.py, it just sleeps forever (never seems to get data?) It seems that maybe the problem is that the producer never terminates, since if I send a finite amount of data then it works fine.

How can I get the data to be received and show up for the consumer? In the real application, the producer is a C++ program I have no control over.

Answer 1

Some old versions of Windows simulated pipes through files (so they were prone to such problems), but that hasn't been a problem in 10+ years. Try adding a

  sys.stdout.flush()

to the producer after the print, and also try to make the producer's stdout unbuffered (by using python -u).

Of course this doesn't help if you have no control over the producer -- if it buffers too much of its output you're still going to wait a long time.

Unfortunately - while there are many approaches to solve that problem on Unix-like operating systems, such as pyexpect, pexpect, exscript, and paramiko, I doubt any of them works on Windows; if that's indeed the case, I'd try Cygwin, which puts enough of a Linux-like veneer on Windows as to often enable the use of Linux-like approaches on a Windows box.

Answer 2

This is about I/O that is bufferized by default with Python. Pass -u option to the interpreter to disable this behavior:

python -u producer.py | python consumer.py

It fixes the problem for me.