Python calling pipe.communicate() in a thread

Question

Using Python 2.6.1 on Mac OS X 10.6.2, I've the following problem:

I have a threaded process (a Thread class), and each of those threads has a pipe (a subprocess.Popen) something likeso:

 from threading import Thread

 cmd = "some_cmd"

 class Worker(Thread):
    def run(self):
       pipe = Popen(cmd,
        stdin=PIPE,
        stdout=PIPE,
        stderr=PIPE)

       out, err = pipe.communicate("some data")

The problem is that the pipe.communicate() code is blocking. Interestingly, when I sent an interrupt (e.g. Ctrl-C KeyboardInterrupt) to the parent process then it unblocks.

Interestingly, when I use class Worker(multiprocessing.Process), the code works just fine.

Any thoughts as to why this is blocking - and how to fix it - would be greatly appreciated.

Thank you.

Answer 1

Using multiple threads and multiple processes will often cause problems, particularly (though not exclusively) on Unix-based systems; I recommend you just avoid mixing the two.

Answer 2

If you call sys.exit() in the main thread thread, other threads will terminate at the next opportunity (on most operating systems). However, if they're in a blocking call (such as communicate()), they will wait until the blocking call completes before terminating. Control-C works because it causes the operating system to interrupt the blocking call.

Generally, the safest approach is to ensure that none of your threads call functions that may block indefinitely. Unfortunately, that often involves writing a lot more code.

In your particular case, you could store the Popen objects in a global set() before calling communicate, and have the main thread call Popen.terminate() on each of them just before exiting. That would cause the child to exit, communicate() to return, and the thread to exit.

Are you setting the thread as a daemon thread?