Starting and Controlling an External Process via STDIN/STDOUT with Python

Question

I need to launch an external process that is to be controlled via messages sent back and forth via stdin and stdout. Using subprocess.Popen I am able to start the process but am unable to control the execution via stdin as I need to.

The flow of what I'm trying to complete is to:

1. Start the external process
2. Iterate for some number of steps
   1. Tell the external process to complete the next processing step by writing a new-line character to it's stdin
   2. Wait for the external process to signal it has completed the step by writing a new-line character to it's stdout
3. Close the external process's stdin to indicate to the external process that execution has completed.

I have come up with the following so far:

process=subprocess.Popen([PathToProcess],stdin=subprocess.PIPE,stdout=subprocess.PIPE);
for i in xrange(StepsToComplete):
    print "Forcing step # %s"%i
    process.communicate(input='\n')

When I run the above code the '\n' is not communicated to the external process, and I never get beyond step #0. The code blocks at process.communicate() and does not proceed any further. I am using the communicate() method incorrectly?

Also how would I implement the "wait until the external process writes a new line" piece of functionality?

Answer 1

process.communicate(input='\n') is wrong. If you will notice from the Python docs, it writes your string to the stdin of the child, then reads all output from the child until the child exits. From doc.python.org:

Popen.communicate(input=None) Interact with process: Send data to stdin. Read data from stdout and stderr, until end-of-file is reached. Wait for process to terminate. The optional input argument should be a string to be sent to the child process, or None, if no data should be sent to the child.

Instead, you want to just write to the child's stdout. Then read from it in your loop.

Something more like:

process=subprocess.Popen([PathToProcess],stdin=subprocess.PIPE,stdout=subprocess.PIPE);
for i in xrange(StepsToComplete):
    print "Forcing step # %s"%i
    process.stdin.write("\n")
    result=process.stdout.readline()

This will do something more like what you want.

Answer 2

You could use Twisted, by using reactor.spawnProcess and LineReceiver.