Broken Pipe from subprocess.Popen.communciate() with stdin

Question

I'm having a strange issue when using subprocess.Popen.communicate(). For background, I want to execute an application from my python script. When I run the program from the command line, I do it like this (UNIX):

$ echo "input text" | /path/to/myapp

From my script, I also want to pipe the input into the application. So, I tried the following. But I get a "broken pipe" error when I try to send the input with communicate():

>>> cmd = ['/path/to/myapp']
>>> p = subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.STDOUT, stdin=subprocess.PIPE)
>>> out,err = p.communicate('input text')
Traceback (most recent call last):
  File "", line 1, in 
  File "/usr/lib/python2.5/subprocess.py", line 670, in communicate
    return self._communicate(input)
  File "/usr/lib/python2.5/subprocess.py", line 1223, in _communicate
    bytes_written = self._write_no_intr(self.stdin.fileno(), buffer(input, input_offset, 512))
  File "/usr/lib/python2.5/subprocess.py", line 1003, in _write_no_intr
    return os.write(fd, s)
OSError: [Errno 32] Broken pipe

To make matters stranger, if I leave out the input data, I don't get any errors. However, this isn't really a good workaround because the application needs input to work.

>>> p = subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.STDOUT, stdin=subprocess.PIPE)
>>> out,err = p.communicate()
>>> print out
[error from myapp regarding lack of input]

Any idea what I'm missing?

Answer 1

Your observation suggests that myapp is terminating without reading (all of the) input. Not knowing anything about myapp, that's hard to confirm, but consider for example

$ echo 'hello world' | tr 'l' 'L'
heLLo worLd

now...:

>>> cmd = ['/usr/bin/tr']
>>> p = subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.STDOUT, stdin=subprocess.PIPE)
>>> out,err = p.communicate('hello world')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python2.5/subprocess.py", line 668, in communicate
    return self._communicate(input)
  File "/usr/lib/python2.5/subprocess.py", line 1218, in _communicate
    bytes_written = self._write_no_intr(self.stdin.fileno(), buffer(input, input_offset, 512))
  File "/usr/lib/python2.5/subprocess.py", line 997, in _write_no_intr
    return os.write(fd, s)
OSError: [Errno 32] Broken pipe

because...:

>>> p = subprocess.Popen(cmd, stdout=subprocess.PIPE, stdin=subprocess.PIPE)
>>> /usr/bin/tr: missing operand
Try `/usr/bin/tr --help' for more information.

and if we fix the bug:

>>> cmd = ['/usr/bin/tr', 'l', 'L']
>>> p = subprocess.Popen(cmd, stdout=subprocess.PIPE, stdin=subprocess.PIPE)
>>> out,err = p.communicate('hello world')>>> print out
heLLo worLd
>>> print err
None

...it fixes everything. What happens if you omit the stderr redirection -- do you perchance see any error messages from myapp...?