making python programs "chat" via pipe

Question

Hi!

I'm trying to make two processes communicate using a pipe. I did this in the parent process:

process = subprocess.Popen(test, shell=True, stdin=subprocess.PIPE, stdout=subprocess.PIPE)

process.stdin.write("4\n");
output = process.stdout.read()
print output

and in the child process:

inp = raw_input()
integer = int(inp)
print integer**2
while(True):
    pass

I would expect the parent process to print 16... Unfortunately, it remains hanging without printing anything. Replacing the infinite loop by a sleep for 5 seconds makes the parent process be idle for 5 seconds and AFTER that print 16. This shows that the parent process only gets the output from the child after it terminated execution.

I'd like to know if it's possible to get input before programs finishes. My idea is to go passing information via this pipe, getting input, processing it, and outputting the result in the pipe, so that the other can continue with processing.

Any help? Thanks,

Manuel

Answer 1

I see a number of possible issues:

a) The child process never actually flushes its output and thus never actually sends its output to the parent.

b) The parent process runs its read() call before the child process has actually sent its output (flushed its output).

c) The parent process does a blocking read() which until EOF which effectively means it waits for the child to exit. The subprocess.Popen docs should mention whether this can be the case.

d) The parent process's read() call waits for a certain number of bytes (e.g. 1024) to arrive before it returns.

Probably having the parent do lots of read(1) calls and reassembling the bytes as they come in will fix the issue. Or using a higher level communication API, e.g. using datagrams instead of parsing byte streams.

Answer 2

You have to use select for this. Like in the communicate method of the subprocess. There's a discussion about this in the python issue tracker.

Answer 3

As suggested by ndim, do the folowing in the parent:

process.stdin.write("4\n")    
process.stdin.flush()    
output = process.stdout.readline()    
print output

You will also need to change the child:

inp = sys.stdin.readline()
integer = int(inp)
sys.stdout.write("%d\n", (integer ** 2,))
sys.stdout.flush()

I used sys.stdin.readline and sys.stdout.write as a matter of style.

I wrote 2 test programs and they worked fine with python-2.6.4-27 on Fedora 13.

Answer 4

THE DEAL: http://code.activestate.com/recipes/440554/