How to defer a Django DB operation from within Twisted?

Question

I have a normal Django site running. In addition, there is another twisted process, which listens for Jabber presence notifications and updates the Django DB using Django's ORM.

So far it works as I just call the corresponding Django models (after having set up the settings environment correctly). This, however, blocks the Twisted app, which is not what I want.

As I'm new to twisted I don't know, what the best way would be to access the Django DB (via its ORM) in a non-blocking way using deferreds.

1. deferredGenerator ?
2. twisted.enterprise.adbapi ? (circumvent the ORM?)
3. ???

If the presence message is parsed I want to save in the Django DB that the user with jid_str is online/offline (using the Django model UserProfile). I do it with that function:

def django_useravailable(jid_str, user_available):
    try:
        userhost = jid.JID(jid_str).userhost()
        user = UserProfile.objects.get(im_jabber_name=userhost)
        user.im_jabber_online = user_available
        user.save()
        return jid_str, user_available
    except Exception, e:
        print e
    raise jid_str, user_available,e

Currently, I invoke it with:

d = threads.deferToThread(django_useravailable, from_attr, user_available)
d.addCallback(self.success)
d.addErrback(self.failure)

Answer 1

"I have a normal Django site running."

Presumably under Apache using mod_wsgi or similar.

If you're using mod_wsgi embedded in Apache, note that Apache is multi-threaded and your Python threads are mashed into Apache's threading. Analysis of what's blocking could get icky.

If you're using mod_wsgi in daemon mode (which you should be) then your Django is a separate process.

Why not continue this design pattern and make your "jabber listener" a separate process.

If you'd like this process to be run any any of a number of servers, then have it be started from init.rc or cron.

Because it's a separate process it will not compete for attention. Your Django process runs quickly and your Jabber listener runs independently.

Answer 2

I have been successful using the method you described as your current method. You'll find by reading the docs that the twisted DB api uses threads under the hood because most SQL libraries have a blocking API.

I have a twisted server that saves data from power monitors in the field, and it does it by starting up a subthread every now and again and calling my Django save code. You can read more about my live data collection pipeline (that's a blog link).

Are you saying that you are starting up a sub thread and that is still blocking?

Answer 3

I have a running Twisted app where I use Django ORM. I'm not deferring it. I know it's wrong, but hadd no problems yet.