Python Twisted and database connections

Question

Hi all,

Our projects at work include synchronous applications (short lived) and asynchronous Twisted applications (long lived). We're re-factoring our database and are going to build an API module to decouple all of the SQL in that module. I'd like to create that API so both synchronous and asynchronous applications can use it. For the synchronous applications I'd like calls to the database API to just return data (blocking) just like using MySQLdb, but for the asynchronous applications I'd like calls to the same API functions/methods to be non-blocking, probably returning a deferred. Anyone have any hints, suggestions or help they might offer me to do this? Thanks in advance, Doug

Answer 1

Have you considered borrowing a page from continuation-passing style? Stackless Python supports continuations directly, if you're using it, and the approach appears to have gained some interest already.

Answer 2

twisted.enterprise.adbapi seems the way to go -- do you think it fails to match your requirements, and if so, can you please explain why?

Answer 3

Within Twisted, you basically want a wrapper around a function which returns a Deferred (such as the Twisted DB layer), waits for it's results, and returns them. However, you can't busy-wait, since that's using up your reactor cycles, and checking for a task to complete using the Twisted non-blocking wait is probably inefficient.

Will inlineCallbacks or deferredGenerator solve your problem? They require a modern Twisted. See the twistedmatrix docs.

def thingummy():
   thing = yield makeSomeRequestResultingInDeferred()
   print thing #the result! hoorj!
thingummy = inlineCallbacks(thingummy)

Another option would be to have two methods which execute the same SQL template, one which uses runInteraction, which blocks, and one which uses runQuery, which returns a Deferred, but that would involve more code paths which do the same thing.

Answer 4

Karl,

Thanks for the feedback. I will take a run at implementing 'thingummy()' and see how that works for me. I also agree, having two code paths through every API method seems like a quick way for things to get broken, but I will think about that one as well.

Alex,

I'm far from a Twisted expert, but trying to get up to speed. I've written an application that used the twisted.enterprise.adapi, which works great in a non-blocking, twisted program. But what I want is to block when an API method from my library is called by an blocking, non-twisted program. Maybe I'm missing something about the twisted.enterprise.adapi.... ?

Thanks again! Doug

Answer 5

All the database libraries I've seen seem to be stubbornly synchronous.

It appears that Twisted.enterprise.abapi solves this problem by using a threads to manage a connection pool and wrapping the underlying database libraries. This is obviously not ideal, but I suppose it would work, but I haven't actually tried it myself.

Ideally there would be some way to have sqlalchemy and twisted integrated. I found this project, nadbapi, which claims to do it, but it looks like it hasn't been updated since 2007.