Multithreading python application hangs while running its threads.

Question

I am trying to create a MainObject which is availible as a DBus service. This MainObject should always stay responsive to other objects/processes and for this non-blocking even while processing its items. for that reason items are processed in a seperate thread one after another (queue-style). You can add items to the MainObject via DBus or CommandLine. I simplified the sample (no dbus, no commandline) to show my problem.

My problem is, that if i reenable 'tt.join()' the application works as expected, but it is blocking to other processes. No wonder, tt.join makes the application wait until the seperate Thread has finished its work. On the other hand, if 'tt.join()' stays disabled, the application does not block to external dbus events, but never comes to 'ThreadTest done!' (look at real output)

What i want is, my expected output but the applications should stay responsive.

#!/usr/bin/python2.5

import gobject
import threading
import re
import time

class ThreadTest(threading.Thread):

  def __init__(self):
    threading.Thread.__init__ (self)    
    print '  ThreadTest created!'

  def run(self):
    print '  ThreadTest running ...'
    time.sleep(1)
    print '  ThreadTest done!'
    return True

class MainObject():
  def __init__(self):
    self.timer = gobject.timeout_add(1000, self.update)
    self.loop  = gobject.MainLoop()
    print 'MainObject created!'

  def update(self):
    print 'MainObject updating ...'
    if self.check_running() == False:
      tt = ThreadTest()
      tt.start()
      #tt.join()
    return True

  def check_running(self):
    running = False
    expr = re.compile('ThreadTest')
    for threadstr in threading.enumerate():
      matches = expr.findall(str(threadstr))
      if matches:
        running = True
    return running  


mo = MainObject()
mo.loop.run()

expected output:

MainObject created!
MainObject updating ...
  ThreadTest created!
  ThreadTest running ...
  ThreadTest done!
MainObject updating ...
  ThreadTest created!
  ThreadTest running ...
  ThreadTest done!
MainObject updating ...
  ThreadTest created!
  ThreadTest running ...
  ThreadTest done!
MainObject updating ...
  ThreadTest created!
  ThreadTest running ...
  ThreadTest done!
MainObject updating ...
  ThreadTest created!
  ThreadTest running ...
  ThreadTest done!

real output:

MainObject created!
MainObject updating ...
  ThreadTest created!
  ThreadTest running ...
MainObject updating ...
MainObject updating ...
MainObject updating ...
MainObject updating ...
MainObject updating ...
MainObject updating ...
MainObject updating ...
MainObject updating ...
MainObject updating ...

Answer 1

Hi -- Threads in python can be a trap -- it is an open problem, actually. The main problem is the GIL - Python's Global Interpreter Lock.

One of the ways they invented to overcome this is the "multiprocessing" module (new in python 2.6) - it preserves the "threading" interface, but actually runs the code in a separate process: You could try jsut replacing htreading by multiprocessing -- but: do all the dbus interaction, GUI, etc.. on a main "thread" - (i.e. a single process) - and trade data with the sub-proccesses (strings, lists, dictionaries, etc)... that would work fine.

Also, I can't figure out why all this regexp Voodo just to check if a given string is present on the return value of threading.enumerate?? Python has even the "in" operator (so you don't even have to use the str's index, or find methods):

you can replace that entire check_running method for:

def check_running(self):
    return 'ThreadTest' in str(threading.enumerate())

Answer 2

The gobject bindings by default aren't multithread-aware. Please try to do the following just after you have imported gobject:

gobject.threads_init()