Popen.communicate() throws OSError: "[Errno 10] No child processes"

Question

I'm trying to start up a child process and get its output on Linux from Python using the subprocess module:

#!/usr/bin/python2.4
import subprocess

p = subprocess.Popen(['ls', '-l', '/etc'],
                   stdout=subprocess.PIPE,
                   stderr=subprocess.PIPE)
out, err = p.communicate()

However, I experience some flakiness: sometimes, p.communicate() would throw

OSError: [Errno 10] No child processes

What can cause this exception? Is there any non-determinism or race condition here that can cause flakiness?

Answer 1

You might be running into the bug mentioned here: http://bugs.python.org/issue1731717

Answer 2

I'm not able to reproduce this on my Python (2.4.6-1ubuntu3). How are you running your script? How often does this occur?

Answer 3

I ran into this problem using Python 2.6.4 which I built into my home directory (because I don't want to upgrade the "built-in" Python on the machine).

I worked around it by replacing subprocess.Popen() with (the deprecated) os.popen3().

Answer 4

Are you intercepting SIGCHLD in the script? If you are then Popen will not run as expected since it relies on it's own handler for that signal.

You can check for SIGCHLD handlers by commenting out the Popen call and then running:

strace python <your_script.py> | grep SIGCHLD

if you see something similar to:

rt_sigaction(SIGCHLD, ...)

then, you are in trouble. You need to disable the handler prior to calling Popen and then resetting it after communicate is done (this might introduce a race conditions so beware).

signal.signal(SIGCHLD, handler)
...
signal.signal(SIGCHLD, signal.SIG_DFL)
'''
now you can go wild with Popen. 
WARNING!!! during this time no signals will be delivered to handler
'''
...
signal.signal(SIGCHLD, handler)

There is a python bug reported on this and as far as I see it hasn't been resolved yet:

http://bugs.python.org/issue9127

Hope that helps.