DBus-Cherrypy merge issue

Question

I'm using python-dbus and cherrypy to monitor USB devices and provide a REST service that will maintain status on the inserted USB devices. I have written and debugged these services independently, and they work as expected.

Now, I'm merging the services into a single application. My problem is: I cannot seem to get both services ( cherrypy and dbus ) to start together. One or the other blocks or goes out of scope, or doesn't get initialized.

I've tried encapsulating each in its own thread, and just call start on them. This has some bizarre issues.

class RESTThread(threading.Thread):
    def __init__(self):
        threading.Thread.__init__(self)

    def run(self):
        cherrypy.config.update({ 'server.socket_host': HVR_Common.DBUS_SERVER_ADDR, 'server.socket_port': HVR_Common.DBUS_SERVER_PORT, })
        cherrypy.quickstart(USBRest())

class DBUSThread(threading.Thread):
    def __init__(self):
        threading.Thread.__init__(self)

    def run(self):
        DBusGMainLoop(set_as_default=True)
        loop = gobject.MainLoop()
        DeviceAddedListener()
        print 'Starting DBus'
        loop.run()

print 'DBus Python Started'
if __name__ == '__main__':
    # Start up REST

    print 'Starting REST'
    rs = RESTThread()
    rs.start()

    db = DBUSThread()
    db.start()

    #cherrypy.config.update({ 'server.socket_host': HVR_Common.DBUS_SERVER_ADDR, 'server.socket_port': HVR_Common.DBUS_SERVER_PORT, })
    #cherrypy.quickstart(USBRest())

    while True:
        x = 1

When this code is run, the cherrypy code doesn't fully initialize. When a USB device is inserted, cherrypy continues to initialize ( as if the threads are linked somehow ), but doesn't work ( doesn't serve up data or even make connections on the port ) I've looked at cherrypys wiki pages but haven't found a way to startup cherrypy in such a way that it inits, and returns, so I can init the DBus stuff an be able to get this out the door.

My ultimate question is: Is there a way to get cherrypy to start and not block but continue working? I want to get rid of the threads in this example and init both cherrypy and dbus in the main thread.

Answer 1

Yes; don't use cherrypy.quickstart. Instead, unpack it:

cherrypy.config.update(conf)
cherrypy.tree.mount(USBREST())
cherrypy.engine.start()

Quickstart does the above, but finishes by calling engine.block(). If your program has some main loop other than CherryPy's, omit the call to engine.block and you should be fine. However, when your foreign main loop terminates, you'll still want to call cherrypy.engine.stop():

loop = gobject.MainLoop()
try:
    loop.run()
finally:
    cherrypy.engine.stop()

There are some other gotchas, like whether CherryPy should handle Ctrl-C and other signals, and whether it should autoreload. Those behaviors are up to you, and are all fairly easy to enable/disable. See the cherrypy.quickstart() source code for some of them.

Answer 2

I figured this out. Apparently, there are a bunch of thread contention problems in glib. If you make an app that has DBusGMainLoop in it, then you cannot create another thread in your app. The new thread blocks immediately when start() is called on it. No amount of massaging will get the new thread to run.

I found a site that had an obscure reference to dbus.mainloop.glib.threads_init(), and how this must be called before initializing a new thread. However a new problem is uncovered when this is tried. An exception is thrown that says g_thread_init() must be called before dbus.mainloop.glib.threads_init() is called. More searching uncovered another obscure reference to gobject.threads_init(). It seemed to fit, so after much experimentation, I discovered the correct sequence.

Here's the solution.

dbus.mainloop.glib.DBusGMainLoop(set_as_default=True)
gobject.threads_init()
dbus.mainloop.glib.threads_init()    
DBUSMAINLOOP = gobject.MainLoop()

print 'Creating DBus Thread'
DBUSLOOPTHREAD = threading.Thread(name='glib_mainloop', target=DBUSMAINLOOP.run)
DBUSLOOPTHREAD.start()

print 'Starting REST'
cherrypy.config.update({ 'server.socket_host': Common.DBUS_SERVER_ADDR, 'server.socket_port': Common.DBUS_SERVER_PORT, })
cherrypy.quickstart(USBRest())

Gosh what a nightmare. Now to make it better.

Answer 3

Hi do you have a sample project for all of this. I have been trying to do something similar to hook into dbus to perform a privileged action using a server daemon. I can get cherrypy working fine, it sends the request over dbus, the action is performed and signal handlers I have registered are invoked, but I can't get them to redirect to a page handler to update the status as the action proceeds, because when they are triggered this is happening in the Dbus thread, so it knows nothing about the cherrypy HTTPredirect exception.