Beginner-level Python threading problems

Question

As someone new to GUI development in Python (with pyGTK), I've just started learning about threading. To test out my skills, I've written a simple little GTK interface with a start/stop button. The goal is that when it is clicked, a thread starts that quickly increments a number in the text box, while keeping the GUI responsive.

I've got the GUI working just fine, but am having problems with the threading. It is probably a simple problem, but my mind is about fried for the day. Below I have pasted first the trackback from the Python interpreter, followed by the code. You can go to http://drop.io/pxgr5id to download it. I'm using bzr for revision control, so if you want to make a modification and re-drop it, please commit the changes. I'm also pasting the code at http://dpaste.com/113388/ because it can have line numbers, and this markdown stuff is giving me a headache.

Update 27 January, 15:52 EST: Slightly updated code can be found here: http://drop.io/threagui/asset/thread-gui-rev3-tar-gz

Traceback

crashsystems@crashsystems-laptop:~/Desktop/thread-gui$ python threadgui.pybtnStartStop clicked
Traceback (most recent call last):
  File "threadgui.py", line 39, in on_btnStartStop_clicked
    self.thread.stop()
  File "threadgui.py", line 20, in stop
    self.join()
  File "/usr/lib/python2.5/threading.py", line 583, in join
    raise RuntimeError("cannot join thread before it is started")
RuntimeError: cannot join thread before it is started
btnStartStop clicked
threadStop = 1
btnStartStop clicked
threadStop = 0
btnStartStop clicked
Traceback (most recent call last):
  File "threadgui.py", line 36, in on_btnStartStop_clicked
    self.thread.start()
  File "/usr/lib/python2.5/threading.py", line 434, in start
    raise RuntimeError("thread already started")
RuntimeError: thread already started
btnExit clicked
exit() called

Code

#!/usr/bin/bash
import gtk, threading

class ThreadLooper (threading.Thread):
    def __init__ (self, sleep_interval, function, args=[], kwargs={}):
        threading.Thread.__init__(self)
        self.sleep_interval = sleep_interval
        self.function = function
        self.args = args
        self.kwargs = kwargs
        self.finished = threading.Event()

    def stop (self):
        self.finished.set()
        self.join()

    def run (self):
        while not self.finished.isSet():
            self.finished.wait(self.sleep_interval)
            self.function(*self.args, **self.kwargs)

class ThreadGUI:
    # Define signals
    def on_btnStartStop_clicked(self, widget, data=None):
        print "btnStartStop clicked"
        if(self.threadStop == 0):
            self.threadStop = 1
            self.thread.start()
        else:
            self.threadStop = 0
            self.thread.stop()
        print "threadStop = " + str(self.threadStop)

    def on_btnMessageBox_clicked(self, widget, data=None):
        print "btnMessageBox clicked"
        self.lblMessage.set_text("This is a message!")
        self.msgBox.show()

    def on_btnExit_clicked(self, widget, data=None):
        print "btnExit clicked"
        self.exit()

    def on_btnOk_clicked(self, widget, data=None):
        print "btnOk clicked"
        self.msgBox.hide()

    def on_mainWindow_destroy(self, widget, data=None):
        print "mainWindow destroyed!"
        self.exit()

    def exit(self):
        print "exit() called"
        self.threadStop = 1
        gtk.main_quit()

    def threadLoop(self):
        # This will run in a thread
        self.txtThreadView.set_text(str(self.threadCount))
        print "hello world"
        self.threadCount += 1

    def __init__(self):
        # Connect to the xml GUI file
        builder = gtk.Builder()
        builder.add_from_file("threadgui.xml")

        # Connect to GUI widgets
        self.mainWindow = builder.get_object("mainWindow")

        self.txtThreadView = builder.get_object("txtThreadView")
        self.btnStartStop = builder.get_object("btnStartStop")
        self.msgBox = builder.get_object("msgBox")
        self.btnMessageBox = builder.get_object("btnMessageBox")
        self.btnExit = builder.get_object("btnExit")
        self.lblMessage  = builder.get_object("lblMessage")
        self.btnOk = builder.get_object("btnOk")

        # Connect the signals
        builder.connect_signals(self)

        # This global will be used for signaling the thread to stop.
        self.threadStop = 1

        # The thread
        self.thread = ThreadLooper(0.1, self.threadLoop, (1,0,-1))
        self.threadCounter = 0

if __name__ == "__main__":
    # Start GUI instance
    GUI = ThreadGUI()
    GUI.mainWindow.show()
    gtk.main()

Answer 1

Generally it's better to avoid threads when you can. It's very difficult to write a threaded application correctly, and even more difficult to know you got it right. Since you're writing a GUI application, it's easier for you to visualize how to do so, since you already have to write your application within an asynchronous framework.

The important thing to realize is that a GUI application is doing a whole lot of nothing. It spends most of its time waiting for the OS to tell it that something has happened. You can do a lot of stuff in this idle time as long as you know how to write long-running code so it doesn't block.

You can solve your original problem by using a timeout; telling your GUI framework to call back some function after a delay, and then resetting that delay or starting another delayed call.

Another common question is how to communicate over the network in a GUI application. Network apps are like GUI apps in that they do a whole lot of waiting. Using a network IO framework (like Twisted) makes it easy to have both parts of your application wait cooperatively instead of competitively, and again alleviates the need for extra threads.

Long-running calculations can be written iteratively instead of synchronously, and you can do your processing while the GUI is idle. You can use a generator to do this quite easily in python.

def long_calculation(param, callback):
    result = None
    while True:
        result = calculate_next_part(param, result)
        if calculation_is_done(result):
            break
        else:
            yield
    callback(result)

Calling long_calculation will give you a generator object, and calling .next() on the generator object will run the generator until it reaches either yield or return. You would just tell the GUI framework to call long_calculation(some_param, some_callback).next when it has time, and eventually your callback will be called with the result.

I don't know GTK very well, so I can't tell you which gobject functions you should be calling. With this explanation, though, you should be able to find the necessary functions in the documentation, or at worst, ask on a relevant IRC channel.

Unfortunately there is no good general-case answer. If you clarify with exactly what you're trying to do, it would be easier to explain why you don't need threads in that situation.

Answer 2

You can't restart a stopped thread object; don't try. Instead, create a new instance of the object if you want to restart it after it's truly stopped and joined.

Answer 3

Threading with PyGTK is bit tricky if you want to do it right. Basically, you should not update GUI from within any other thread than main thread (common limitation in GUI libs). Usually this is done in PyGTK using mechanism of queued messages (for communication between workers and GUI) which are read periodically using timeout function. Once I had a presentation on my local LUG on this topic, you can grab example code for this presentation from Google Code repository. Have a look at MainWindow class in forms/frmmain.py, specially for method _pulse() and what is done in on_entry_activate() (thread is started there plus the idle timer is created).

def on_entry_activate(self, entry):
    text = entry.get_text().strip()
    if text:
        store = entry.get_completion().get_model()
        if text not in [row[0] for row in store]:
            store.append((text, ))
        thread = threads.RecommendationsFetcher(text, self.queue)# <- 1
        self.idle_timer = gobject.idle_add(self._pulse)# <- 2
        tv_results = self.widgets.get_widget('tv_results')
        model = tv_results.get_model()
        model.clear()
        thread.setDaemon(True)# <- 3
        progress_update = self.widgets.get_widget('progress_update')
        progress_update.show()
        thread.start()# <- 4

This way, application updates GUI when is "idle" (by GTK means) causing no freezes.

• 1: create thread
• 2: create idle timer
• 3: daemonize thread so the app can be closed without waiting for thread completion
• 4: start thread

Answer 4

I've played with different tools to help clean up the work with threads, idle processing, etc.

make_idle is a function decorator that allows you to run a task in the background cooperatively. This is a good middle ground between something short enough to run once in the UI thread and not affect the responsiveness of the app and doing a full out thread in special synchronization. Inside the decorated function you use "yield" to hand the processing back over to the GUI so it can remain responsive and the next time the UI is idle it will pick up in your function where you left off. So to get this started you just call idle_add to the decorated function.

def make_idler(func):
    """
    Decorator that makes a generator-function into a function that will
continue execution on next call
    """
    a = []

    @functools.wraps(func)
    def decorated_func(*args, **kwds):
        if not a:
            a.append(func(*args, **kwds))
        try:
            a[0].next()
            return True
        except StopIteration:
            del a[:]
            return False

    return decorated_func

If you need to do a bit more processing, you can use a context manager to lock the UI thread whenever needed to help make the code a bit safer

@contextlib.contextmanager
def gtk_critical_section():
    gtk.gdk.threads_enter()
    try:
        yield
    finally:
        gtk.gdk.threads_leave()

with that you can just

with gtk_critical_section():
    ... processing ...

I have not finished with it yet, but in combining doing things purely in idle and purely in a thread, I have a decorator (not tested yet so not posted) that you can tell it whether the next section after the yield is to be run in the UI's idle time or in a thread. This would allow one to do some setup in the UI thread, switch to a new thread for doing background stuff, and then switch over to the UI's idle time to do cleanup, minimizing the need for locks.

Answer 5

I haven't looked in detail on your code. But I see two solutions to your problem:

Don't use threads at all. Instead use a timeout, like this:

import gobject

i = 0
def do_print():
    global i
    print i
    i += 1
    if i == 10:
        main_loop.quit()
        return False
    return True

main_loop = gobject.MainLoop()
gobject.timeout_add(250, do_print)
main_loop.run()

When using threads, you must make sure that your GUI code is only called from one thread at the same time by guarding it like this:

import threading
import time

import gobject
import gtk

gtk.gdk.threads_init()

def run_thread():
    for i in xrange(10):
        time.sleep(0.25)
        gtk.gdk.threads_enter()
        # update the view here
        gtk.gdk.threads_leave()
    gtk.gdk.threads_enter()
    main_loop.quit()
    gtk.gdk.threads_leave()

t = threading.Thread(target=run_thread)
t.start()
main_loop = gobject.MainLoop()
main_loop.run()