How to design an email system?

Question

I am working for a company that provides customer support to its clients. I am trying to design a system that would send emails automatically to clients when some event occurs. The system would consist of a backend part and a web interface part. The backend will handle the communication with a web interface (which will be only for internal use to change the email templates) and most important it will check some database tables and based on those results will send emails ... lots of them.

Now, I am wondering how can this be designed so it can be made scalable and provide the necessary performance as it will probably have to handle a few thousands emails per hours (this should be the peek). I am mostly interested about how would this kind of architecture should be thought in order to be easily scaled in the future if needed.

Python will be used on the backend with Postgres and probably whatever comes first between a Python web framework and GWT on the frontend (which seems the simplest task).

Answer 1

This sound to me, that you're trying to optimize for batch processing, where the heat doenst happen on the web interface but in the backend. This also sounds a job for a queuing architecture.

Amazon offers queuing systems for instance if you really need massive scale. So you can add multiple machines on your side to deliver the messages as eMails. So you allow one machines only taking perhaps 100 messages from the queue at one time.

The pattern with eMail systems should be asychonous, so have a look at other asynchonous archictures if you dont like queues.

Answer 2

This is a real good candidate for using some off the shelf software. There are any number of open-source mailing list manager packages around; they already know how to do the mass mailings. It's not completely clear whether these mailings would go to the same set of people each time; if so, get any one of the regular mailing list programs.

If not, the easy answer is

$ mail address -s subject < file

once per mail.

By the way, investigate the policies of whoever is upstream from you on the net. Some ISPs see lots of mails as probable spam, and may surprise you by cutting off or metering your internet access.

Answer 3

A few thousand emails per hour isn't really that much, as long as your outgoing mail server is willing to accept them in a timely manner.

I would send them using a local mta, like postfix, or exim (which would then send them through your outgoing relay if required). That service is then responsible for the mail queues, retries, bounces, etc. If your looking for more "mailing list" features, try adding mailman into the mix. It's written in python, and you've probably seen it, as it runs tons of internet mailing lists.

Answer 4

You might want to try Twisted Mail for implementing your own backend in pure Python.

Answer 5

You might want to check out Lamson, a state machine-based e-mail server written in Python that should be able to do what you have described. It's written by Zed Shaw, and he blogged about it recently here.