How can I find out why subprocess.Popen wait() waits forever if stdout=PIPE?

Question

I have a program that writes to stdout and possibly stderr. I want to run it from python, capturing the stdout and stderr. My code looks like:

from subprocess import *

p = Popen( exe, shell=TRUE, stdout=PIPE, stderr=PIPE )
rtrncode = p.wait()

For a couple of programs, this works fine, but when I added a new one, the new one hangs forever. If I remove stdout=PIPE, the program writes its output to the console and finishes and everything is fine. How can I determine what's causing the hang?

Using python 2.5 on Windows XP. The program does not read from stdin nor does it have any kind of user input (i.e. "hit a key").

Answer 1

When a pipe's buffer fills up (typically 4KB or so), the writing process stops until a reading process has read some of the data in question; but here you're reading nothing until the subprocess is done, hence the deadlock. The docs on wait put it very clearly indeed:

Warning This will deadlock if the child process generates enough output to a stdout or stderr pipe such that it blocks waiting for the OS pipe buffer to accept more data. Use communicate() to avoid that.

If you can't use communicate for some reason, have the subprocess write to a temporary file, and then you can wait and read that file when it's ready -- writing to a file, instead of to a pipe, does not risk deadlock.

Answer 2

Take a look at the docs. It states that you shouldn't use wait as it can cause a dead lock. Try using communicate.