Need a workaround: Python's select.select() doesn't work with subprocess' stdout?

Question

From within my master python program, I am spawning a child program with this code:

child = subprocess.Popen(..., stdout=subprocess.PIPE, stdin=subprocess.PIPE)

FWIW, the child is a PHP script which needs to communicate back and forth with the python program.

The master python program actually needs to listen for communication from several other channels - other PHP scripts spawned using the same code, or socket objects coming from socket.accept(), and I would like to use select.select() as that is the most efficient way to wait for input from a variety of sources.

The problem I have, is that select.select() under Windows does not work with the subprocess' stdout file descriptor (this is documented), and it looks I will be forced to:

• A) Poll the PHP scripts to see if they have written anything to stdout. (This system needs to be very responsive, I would need to poll at least 1,000 times per second!)
• B) Have the PHP scripts connect to the master process and communicate via sockets instead of stdout/stdin.

I will probably go with solution (B), because I can't bring myself to make the system poll at such a high frequency, but it seems a sad waste of resources to reconnect with sockets when stdout/stdin would have done just fine.

Is there some alternative solution which would allow me to use stdout and select.select()?

Answer 1

Unfortunately, many uses of pipes on Windows don't work as nicely as they do on Unix, and this is one of them. On Windows, the better solution is probably to have your master program spawn threads to listen to each of its subprocesses. If you know the granularity of data that you expect back from your subprocess, you can do blocking reads in each of your threads, and then the thread will come alive when the IO unblocks.

Alternatively, (I have no idea if this is viable for your project), you could look into using a Unix-like system, or a Unix-like layer on top of Windows (e.g. Cygwin), where select.select() will work on subprocess pipes.