Python Multiprocessing exit error

Question

Hey everyone I am seeing this when I press Ctrl-C to exit my app

Error in atexit._run_exitfuncs:
Traceback (most recent call last):
  File "/usr/lib/python2.6/atexit.py", line 24, in _run_exitfuncs
    func(*targs, **kargs)
  File "/usr/lib/python2.6/multiprocessing/util.py", line 269, in _exit_function
    p.join()
  File "/usr/lib/python2.6/multiprocessing/process.py", line 119, in join
    res = self._popen.wait(timeout)
  File "/usr/lib/python2.6/multiprocessing/forking.py", line 117, in wait
    return self.poll(0)
  File "/usr/lib/python2.6/multiprocessing/forking.py", line 106, in poll
    pid, sts = os.waitpid(self.pid, flag)
OSError: [Errno 4] Interrupted system call
Error in sys.exitfunc:
Traceback (most recent call last):
  File "/usr/lib/python2.6/atexit.py", line 24, in _run_exitfuncs
    func(*targs, **kargs)
  File "/usr/lib/python2.6/multiprocessing/util.py", line 269, in _exit_function
    p.join()
  File "/usr/lib/python2.6/multiprocessing/process.py", line 119, in join
    res = self._popen.wait(timeout)
  File "/usr/lib/python2.6/multiprocessing/forking.py", line 117, in wait
    return self.poll(0)
  File "/usr/lib/python2.6/multiprocessing/forking.py", line 106, in poll
    pid, sts = os.waitpid(self.pid, flag)
OSError: [Errno 4] Interrupted system call

I am using twisted on top of my own stuff,

I registered the signal Ctrl-C with the following code

    def sigHandler(self, arg1, arg2):
        if not self.backuped:
            self.stopAll()
        else:
            out('central', 'backuped ALREADY, now FORCE exiting')
            exit()


    def stopAll(self):
        self.parserM.shutdown()
        for each in self.crawlM:
            each.shutdown()
        self.backup()
        reactor.stop()

and when they signal others to shutdown, it tries to tell them to shutdown nicely through

exit = multiprocessing.Event()
def shutdown(self):
    self.exit.set()

where all my processes are in some form,

def run(self):
    while not self.exit.is_set():
        do something
    out('crawler', 'crawler exited sucessfully')

Any idea what this error is? I only get it when I have more than one instance of a particular thread.

Answer 1

Has ANYONE ever encountered this error?

Answer 2

This is related to interactions OS system calls, signals and how it's handled in the multiprocessing module. I'm not really sure if it's a bug or a feature, but it's in somewhat tricky territory as it's where python meets the os.

The problem is that multiprocessing is blocking on waitpid until the child it's waiting for has terminated, however, since you've installed a signal-handler for SIGINT and your program gets this signal, it interrupts the system-call to execute your signal-handler, and waitpid exits indicated that it were interrupted by a signal. The way python handles this case is by exceptions.

As a workaround, you can enclose the offending section(s) in a while-loop and try/catch blocks like this, either around where you wait for threads to finish, or subclass multiprocessing.Popen:

import errno
from multiprocessing import Process

p = Processing( target=func, args=stuff )
p.start()
notintr = False
while not notintr:
  try:
    p.join() # "Offending code"
    notintr = True
  except OSError, ose:
    if ose.errno != errno.EINTR:
      raise ose

For mucking about with multiprocessing.forking.Popen you'd have to do something like this:

import errno
from multiprocessing import Process
from multiprocessing.forking import Popen
import os

# see /path/to/python/libs/multiprocessing/forking.py
class MyPopen(Popen):
  def poll(self, flag=os.WNOHANG): # from forking.py
    if self.returncode is None: # from forking.py
      notintr = False
      while not notintr:
        try:
          pid, sts = os.waitpid(self.pid, flag) # from forking.py
          notintr = True
        except OSError, ose:
          if ose.errno != errno.EINTR:
            raise ose
      # Rest of Popen.poll from forking.py goes here

p = Process( target=func args=stuff )
p._Popen = p
p.start()
p.join()

Answer 3

I was seeing this, but it went away when I overrode signal handlers with my own. Use reactor.run(installSignalHandlers=False) and define your own functions for SIGINT, SIGTERM, etc.