Python: How can I use Twisted as the transport for SUDS?

Question

I have a project that is based on Twisted used to communicate with network devices and I am adding support for a new vendor (Citrix NetScaler) whose API is SOAP. Unfortunately the support for SOAP in Twisted still relies on SOAPpy, which is badly out of date. In fact as of this question (I just checked), twisted.web.soap itself hasn't even been updated in 21 months!

I would like to ask if anyone has any experience they would be willing to share with utilizing Twisted's superb asynchronous transport functionality with SUDS. It seems like plugging in a custom Twisted transport would be a natural fit in SUDS' Client.options.transport, I'm just having a hard time wrapping my head around it.

I did come up with a way to call the SOAP method with SUDS asynchronously by utilizing twisted.internet.threads.deferToThread(), but this feels like a hack to me.

Here is an example of what I've done, to give you an idea:

# netscaler is a module I wrote using suds to interface with NetScaler SOAP
# Source: http://bitbucket.org/jathanism/netscaler-api/src
import netscaler
import os
import sys
from twisted.internet import reactor, defer, threads

# netscaler.API is the class that sets up the suds.client.Client object
host = 'netscaler.local'
username = password = 'nsroot'
wsdl_url = 'file://' + os.path.join(os.getcwd(), 'NSUserAdmin.wsdl')
api = netscaler.API(host, username=username, password=password, wsdl_url=wsdl_url)

results = []
errors = []

def handleResult(result):
    print '\tgot result: %s' % (result,)
    results.append(result)

def handleError(err):
    sys.stderr.write('\tgot failure: %s' % (err,))
    errors.append(err)

# this converts the api.login() call to a Twisted thread.
# api.login() should return True and is is equivalent to:
# api.service.login(username=self.username, password=self.password)
deferred = threads.deferToThread(api.login)
deferred.addCallbacks(handleResult, handleError)

reactor.run()

This works as expected and defers return of the api.login() call until it is complete, instead of blocking. But as I said, it doesn't feel right.

Thanks in advance for any help, guidance, feedback, criticism, insults, or total solutions.

Answer 1

The default interpretation of transport in the context of Twisted is probably an implementation of twisted.internet.interfaces.ITransport. At this layer, you're basically dealing with raw bytes being sent and received over a socket of some sort (UDP, TCP, and SSL being the most commonly used three). This isn't really what a SUDS/Twisted integration library is interested in. Instead, what you want is an HTTP client which SUDS can use to make the necessary requests and which presents all of the response data so that SUDS can determine what the result was. That is to say, SUDS doesn't really care about the raw bytes on the network. What it cares about is the HTTP requests and responses.

If you examine the implementation of twisted.web.soap.Proxy (the client part of the Twisted Web SOAP API), you'll see that it doesn't really do much. It's about 20 lines of code that glues SOAPpy to twisted.web.client.getPage. That is, it's hooking SOAPpy in to Twisted in just the way I described above.

Ideally, SUDS would provide some kind of API along the lines of SOAPpy.buildSOAP and SOAPpy.parseSOAPRPC (perhaps the APIs would be a bit more complicated, or accept a few more parameters - I'm not a SOAP expert, so I don't know if SOAPpy's particular APIs are missing something important - but the basic idea should be the same). Then you could write something like twisted.web.soap.Proxy based on SUDS instead. If twisted.web.client.getPage doesn't offer enough control over the requests or enough information about the responses, you could also use twisted.web.client.Agent instead, which is more recently introduced and offers much more control over the whole request/response process. But again, that's really the same idea as the current getPage-based code, just a more flexible/expressive implementation.

Having just looked at the API documentation for Client.options.transport, it sounds like a SUDS transport is basically an HTTP client. The problem with this kind of integration is that SUDS wants to send a request and then be able to immediately get the response. Since Twisted is largely based on callbacks, a Twisted-based HTTP client API can't immediately return a response to SUDS. It can only return a Deferred (or equivalent).

This is why things work better if the relationship is inverted. Instead of giving SUDS an HTTP client to play with, give SUDS and an HTTP client to a third piece of code and let it orchestrate the interactions.

It may not be impossible to have things work by creating a Twisted-based SUDS transport (aka HTTP client), though. The fact that Twisted primarily uses Deferred (aka callbacks) to expose events doesn't mean that this is the only way it can work. By using a third-party library such as greenlet, it's possible to provide a coroutine-based API, where a request for an asynchronous operation involves switching execution from one coroutine to another, and events are delivered by switching back to the original coroutine. There is a project called corotwine which can do just this. It may be possible to use this to provide SUDS with the kind of HTTP client API it wants; however, it's not guaranteed. It depends on SUDS not breaking when a context switch is suddenly inserted where previously there was none. This is a very subtle and fragile property of SUDS and can easily be changed (unintentionally, even) by the SUDS developers in a future release, so it's probably not the ideal solution, even if you can get it to work now (unless you can get cooperation from the SUDS maintainers in the form of a promise to test their code in this kind of configuration to ensure it continues to work).

As an aside, the reason Twisted Web's SOAP support is still based on SOAPpy and hasn't been modified for nearly two years is that no clear replacement for SOAPpy has ever shown up. There have been many contenders (http://stackoverflow.com/questions/206154/whats-the-best-soap-client-library-for-python-and-where-is-the-documentation-fo covers several of them). If things ever settle down, it may make sense to try to update Twisted's built-in SOAP support. Until then, I think it makes more sense to do these integration libraries separately, so they can be updated more easily and so Twisted itself doesn't end up with a big pile of different SOAP integration that no one wants (which would be worse than the current situation, where there's just one SOAP integration module that no one wants).