BoundedSemaphore hangs in threads on KeyboardInterrupt

Question

If you raise a KeyboardInterrupt while trying to acquire a semaphore, the threads that also try to release the same semaphore object hang indefinitely.

Code:

import threading
import time

def worker(i, sema):
    time.sleep(2)
    print i, "finished"
    sema.release()


sema = threading.BoundedSemaphore(value=5)
threads = []
for x in xrange(100):
    sema.acquire()
    t = threading.Thread(target=worker, args=(x, sema))
    t.start()
    threads.append(t)

Start this up and then ^C as it is running. It will hang and never exit.

0 finished
3 finished
1 finished
2 finished
4 finished
^C5 finished
Traceback (most recent call last):
  File "/tmp/proof.py", line 15, in <module>
    sema.acquire()
  File "/System/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/threading.py", line 290, in acquire
    self.__cond.wait()
  File "/System/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/threading.py", line 214, in wait
    waiter.acquire()
KeyboardInterrupt
6 finished
7 finished
8 finished
9 finished

How can I get it to let the last few threads die natural deaths and then exit normally? (which it does if you don't try to interrupt it)

Answer 1

You can use the signal module to set a flag that tells the main thread to stop processing:

import threading
import time
import signal
import sys

sigint = False

def sighandler(num, frame):
  global sigint
  sigint = True

def worker(i, sema):
  time.sleep(2)
  print i, "finished"
  sema.release()

signal.signal(signal.SIGINT, sighandler)
sema = threading.BoundedSemaphore(value=5)
threads = []
for x in xrange(100):
  sema.acquire()
  if sigint:
    sys.exit()
  t = threading.Thread(target=worker, args=(x, sema))
  t.start()
  threads.append(t)

Answer 2

In this case, it looks like you might just want to use a thread pool to control the starting and stopping of your threads. You could use Chris Arndt's threadpool library in a manner something like this:

pool = ThreadPool(5)
try:
    # enqueue 100 worker threads
    pool.wait()
except KeyboardInterrupt, k:
    pool.dismiss(5)
    # the program will exit after all running threads are complete