How do I handle exceptions when using threading and Queue?

Question

If I have a program that uses threading and Queue, how do I get exceptions to stop execution? Here is an example program, which is not possible to stop with ctrl-c (basically ripped from the python docs).

from threading import Thread
from Queue import Queue
from time import sleep

def do_work(item):
    sleep(0.5)
    print "working" , item

def worker():
        while True:
            item = q.get()
            do_work(item)
            q.task_done()

q = Queue()

num_worker_threads = 10

for i in range(num_worker_threads):
     t = Thread(target=worker)
    # t.setDaemon(True)
     t.start()

for item in range(1, 10000):
    q.put(item)

q.join()       # block until all tasks are done

Answer 1

The simplest way is to start all the worker threads as daemon threads, then just have your main loop be

while True:
    sleep(1)

Hitting Ctrl+C will throw an exception in your main thread, and all of the daemon threads will exit when the interpreter exits. This assumes you don't want to perform cleanup in all of those threads before they exit.

A more complex way is to have a global keep_going variable:

keep_going = True
def worker():
    while keep_going:
        try:
            item = q.get(timeout=1)
            do_work(item)
        except Empty:      # import this from the Queue module
            pas

Then your main loop can set keep_going to False when it gets a KeyboardInterrupt

try:
    while True:
        sleep(1)
except:
    keep_going = False

This lets your worker threads finish what they're doing you want instead of just having every worker thread be a daemon and exit in the middle of execution. You can also do whatever cleanup you want.

Note that this example doesn't make use of q.join() - this makes things more complex, though you can still use it. If you do then your best bet is to use signal handlers instead of exceptions to detect KeyboardInterrupts. For example:

from signal import signal, SIGINT
def stop(signum, frame):
    global keep_going
    keep_going = False
signal(SIGINT, stop)

This lets you define what happens when you hit Ctrl+C without affecting whatever your main loop is in the middle of. So you can keep doing q.join() without worrying about being interrupted by a Ctrl+C. Of course, with my above examples, you don't need to be joining, but you might have some other reason for doing so.

Answer 2

A (possibly) offtopic note:

(...)
for item in range(1, 10000):
    q.put(item)
(...)

You could use xrange instead of range here (unless You use python3000). You will save some cpu and memory by doing so. More on xrange can be found here.