Can you only communicate once with a subprocess?

Question

communicate's documentation says:

Interact with process: Send data to stdin. Read data from stdout and stderr, until end-of-file is reached. Wait for process to terminate.

What do you do if you need to send input to a process more than once ? For example, I spawn a process, send it some data, the process does something with that, returns some output, and then I have to send input again? How do I handle that?

Answer 1

Then you can't use .communicate(). You can either poll the streams, use select or some other way that allows you to listen to FD changes (both gtk and Qt have tools for that, for example).

Answer 2

Take a look at Doug Hellman's Python Module of the Week writeup about subprocess: http://blog.doughellmann.com/2007/07/pymotw-subprocess.html

(Search down until you see "repeater.py").

There you will find an example of how to send and receive input/output to a process.

Answer 3

Don't use subprocess for this, you'll enter a world of pain to do with buffering.

I recommend pexpect for this purpose - it works very well. Unfortunately it doesn't work under windows at the moment, though I did hear about a port (which I can't find any more).

Answer 4

Here's a module I wrote. Make sure to use that -u argument to avoid buffering problems:

import os
import pickle
import subprocess
from subprocess import PIPE
import struct
import builtins
def call_thru_stream(stream,funcname,*args,**kwargs):
    """Used for calling a function through a stream and no return value is required. It is assumed
    the receiving program is in the state where it is expecting a function."""
    transmit_object(stream,(funcname,args,kwargs))


def function_thru_stream(in_stream,out_stream,funcname,*args,**kwargs):
    """Used for calling a function through a stream where a return value is required. It is assumed
    the receiving program is in the state where it is expecting a function."""
    transmit_object(in_stream,(funcname,args,kwargs))
    return receive_object(out_stream)

#--------------------- Object layer ------------------------------------------------------------

def transmit_object(stream,obj):
    """Transmits an object through a binary stream"""
    data=pickle.dumps(obj,2)#Uses pickle version 2 for compatibility with 2x
    transmit_atom(stream,data)


def receive_object(stream):
    """Receive an object through a binary stream"""
    atom=receive_atom(stream)
    return pickle.loads(atom)

#--------------------- Atom layer --------------------------------------------------------------
def transmit_atom(stream, atom_bytes):
    """Used for transmitting a some bytes which should be treated as an atom through
    a stream. An integer indicating the size of the atom is appended to the start."""
    header=struct.pack("=L",len(atom_bytes))
    stream.write(header)
    stream.write(atom_bytes)
    stream.flush()


def receive_atom(stream):
    """Read an atom from a binary stream and return the bytes."""
    input_len=stream.read(4)
    l=struct.unpack("=L",input_len)[0]
    return stream.read(l)