Would twisted be a good choice for building a multi-threaded server?

Question

I need to pull from hundreds of pop3 email accounts, and i want to build a robust server to do this.

Would twisted be a good choice for this type of project?

Right now a simple prototype would be to pull from a single pop3 account, then it would pull from many but it would be a serialized process.

I want to create a server that has multiple threads so it can do things at the same time.

Answer 1

It is a good choice for a server but from your description you are acutally looking for a multithreaded POP client.

Twisted is made for reacting to events like incoming requests, you need to send requests, so in this case I fear twisted to be of limited value.

Answer 2

Considering that the majority of your POP3 activity is going to be network I/O, this is where Twisted excels. You're not really threading so much as performing event-based asynchronous socket operations, which is the crowning glory of Twisted.

So, yes, Twisted would be a good choice for this type of project. It can do client and server operations equally well and it is almost trivial to spin up a new async TCP client and it already has a POP3 TCP Client available by default.

Answer 3

Twisted is an event-driven networking framework written in Python. It builds heavily on asynchronous and non-blocking features and is best conceived to develop networking applications that utilizes these. It has thread support for use cases where you can not provide for asynchronous non-blocking I/O. This is based on the fact that most of time is spent waiting in network I/O operations.

The two model that exploits this is threading model where you create multiple threads, each accomplishing a single task or a single process that uses non-blocking I/O to accomplish multiple task in a single process by interleaving multiple tasks. Twisted is very suitable for the second model.

Non-Blocking model

+--------------------------+
|task1 | wait period | comp|
+--------------------------+
       +--------------------------+
       |task2 | wait period | comp|
       +--------------------------+

You can develop a very robust server with Twisted and it has POP3 / IMAP support.

There is an example of how to build pop3 client with twisted.

Answer 4

A word of caution with twisted, while twisted is very robust I've found that spinning up a hundred threads using the code examples available in documentation is a recipe for race conditions and deadlocks. My suggestion is try twisted but have the stdlib multithreading module waiting in the wings if twisted becomes unmanageable. I have had good success with a producer consumer model using the aforementioned library.