Best practices for sending automated daily emails from web service

Question

I am running a web service that currently sends confirmation emails out to new users via the gmail smtp servers. As I'm only getting a few new users each day, this hasn't been a problem.

I've recently added new features to the webapp that will require a customized message to be sent out to each user every day. Think of this as similar to the regular messages LinkedIn sends out that give you a status report on the activity in your network. Every user's message will be different. With thousands of users, this means thousands of unique messages will be sent each day.

Edit: I've since found that these types of email are called "transactional or relationship messages". Spamtacular has a good article on differentiating between marketing and transactional email.

I don't think using gmail's smtp servers will cut it anymore, but I don't know that for sure. I don't know what gmail's maximum outgoing messages per account is (it might be 100/day), but they limit outgoing mail to 500 recipients per message. I'm not sending a single message to 500 recipients, but I'm going to be sending 1000's of customized messages with each recipient getting one per day.

I'm interested to learn any best practices for doing this (especially for Java-based webapps). Here are some of my thoughts and concerns on it:

• Should I set up my own outgoing mail server? If I do this, it seems like I'll have all sorts of other issues to worry about, such as preventing mail server abuse, monitoring bounces, allowing ways to opt-out of emails, etc. Are there any tools or services to help with this? Maybe something like OpenEMM or a services like MailChimp? But those seem focused more toward email marketing campaigns.
• I don't think I should have the webapp itself handle sending emails as it currently is for new user signups. I'm thinking I should setup a separate messaging server that can access the same backend/datastore as the webapp. Thoughts on this?
• Should I consider setting up some sort of message queueing service to help with this, such as JMS, RabbitMQ, ActiveMQ, etc.?
• Do I need to provide users a way to opt-out? Do I need to flag these as bulk messages? I don't really consider these email marketing messages, but I'm unsure what is considered appropriate or proper netiquette.

Any advice is appreciated. I'm also very interested in open source tools or web services that simplify things and could help me to ramp up as quickly as possible.

Thanks!

Answer 1

With regard to your first question, yes, you should set up your own mail server. Using gmail to do this might work for a while, but they are likely to shut you down in short order when they see this kind of activity. You could sign up for a business account and use app engine to send messages. Here's a link with information about mail quotas for that service.

Regarding your second and third questions, It would be a good idea to have messages queued by the web app and sent out by a centralized service rather than having the app send out the messages on its own.

Usually I would just use a database table as a queue - the web app inserts rows for each message it wants to send. A service/scheduled task app would grab new messages out of the table and send them off. This gives you lots of flexibility if you want to switch mail servers later, better reliability if the mail server is down, easier diagnostics if there are problems with recipients not getting messages, and the ability to resend messages. As for using JMS/MQ to do this - probably not necessary. IMO a database table used as a queue would give you more flexibility here than an actualy JMS-based queue system.

As for opt outs, YES - you should give people a way to opt out. I don't think you need to flag the messages as bulk though.

Answer 2

On the architecture side of things I would definitely consider decoupling the sending of the emails from the main service via some form of asynchronous message queuing (or facsimile thereof using database as an intermediary). Another benefit of this approach is that if the SMPT server\network is down you could build in retry semantics, additionally for future scalability you could implement multiple mail senders reading from the same queue or implement sending throttling or scheduling (i.e send n messages per hour), etc etc.