Who stops my threads?

Question

I have some threads fishing into a queue for jobs, something like this:

class Worker(Thread):
    [...]
    def run(self):
        while not self.terminated:
            job = myQueue.get_nowait()
            job.dosomething()
            sleep(0.5)

Now, self.terminated is just a bool value I use to exit the loop but, this is the problem, several times in a day they stop working without my intervention. All of them but one: the application starts with, lets say, 5 working threads and at random time I check them and one only is working. All the others have both Thread_initialized and Thread_stopped fields true. Threads and jobs does not interact with each other. What I should look for?

PS: I understand it's really hard to try to figure out the issue without the actual code, but it's huge.

UPDATE: actually Queue.Empty is the only exception trapped - guess I believed to let all the jobs' internal errors to propagate without kill the threads eheh - so I'm going to block all the exceptions and see...

Answer 1

stackoverflow? :)

Answer 2

As example, an exception inside the loop will stop the thread.

Why do you use get_nowait() and not get()? What if the Queue is empty?

Answer 3

GIL? at one time only an interpreter is doing the job if you want true parallelization you must use multiprocessing see http://docs.python.org/library/multiprocessing.html

Answer 4

If that is the actual code it's pretty obvious: myQueue.get_nowait() raises an Exception (Empty) when the queue is empty!

Answer 5

I have two suggestions.

1) get_nowait() will raise a Queue.Empty exception if no items are available. Make sure exceptions aren't killing your threads.

2) Use get() instead. Put a None in your queue to signal the thread to exit instead of the boolean flag. Then you don't need a half second sleep and you'll process items faster.

def run(self):
    while True:
        job = queue.get()
        if job:
            try:
                job.do_something()
            except Exception as e:
                print e
        else: # exit thread when job is None
            break