Multiple communication channels with Twisted Python

Question

I am currently researching the Twisted framework as a way of implementing a network-based backup application, and I would like to achieve something that I cannot find any examples of on the net.

I plan to implement the system using the Perspective Broker, but I will also need a way of transferring binary files from the client to the server. I would like to be able to call a method on the PB, and then use some sort of UID to send the file over a separate data channel.

The reason for having these two separate communication channels is down to the fact that I would like to make the client multi-threaded (one thread scanning the directory tree, while another thread transfers the changed files to the server).

Is this possible with Twisted? I have read that having multiple threads calling methods on a reactor is bad news, so is this architecture doomed to failure?

I would appreciate any pointers in the right direction, as I mentioned I am still researching the possibilities - but I plan to use Django for this project, so Python is a must.

Answer 1

The reason for having these two separate communication channels is down to the fact that I would like to make the client multi-threaded (one thread scanning the directory tree, while another thread transfers the changed files to the server).

This reasoning doesn't follow. You can use a single protocol running over a single socket just fine, even if you have a thread wandering the filesystem looking for work to do.

There may be other reasons to want to send file data differently than you send metadata or other structured data between the client and server, though. However, the main one that comes to mind is that you might not want to force commands to wait for files to be completed, and this issue is relieved by PB's FilePager class.

The main thing to remember if you're going to have threads in a Twisted-using application is that whenever you want to invoke a Twisted API from any thread ''except'' the thread in which the reactor is running you must use reactor.callFromThread (or an API built solely on that method, such as twisted.internet.threads.blockingCallFromThread).

callFromThread sends some work (in the form of an object to call) to the reactor thread where the reactor will arrange to call it "soon". Any other Twisted API you invoke from the wrong thread will have undefined results.