What methods exist for setting up a large email notification system?

Question

My company has a website built with PHP. We use the built-in PHP email functionality to send thousands of emails to subscribers on a daily basis.

This is a terrible idea. It chokes out our server, and takes hours to complete the whole batch.

Now I've looked at mass mailing services like MailChimp (which would replace our current system of sending the same email to many people), but what I think I'd really like to do is to set up a somewhat-sophisticated notification system.

Rather than send a mass email to each person each time something important happens, I'd like clients to be able to customize the rate and content of the emails that they receive.

Even using this new idea, we're talking about A LOT of emails being sent.

So my question is very specific: I have a rough idea of how to build the system internally, but what is the best way to send out all of these emails?

Bullet points to consider:

• Sometimes emails' contents are identical across recipients, but many of them will be customized per-user (they choose what they get notified about, and sometimes it is aggregated).
• I want a system that is not going to choke the server, and will complete in a decent amount of time. I don't mind going with a third-party service (even a paid one) if that is what it is going to take.
• The system should hook into PHP easily, or the API or whatever should be relatively easy for me to call from your typical web server.
• We have a dedicated server and full control over it (so we can install apps, services, whatever).
• Any kind of detailed tracking information (opens, clicks, etc) is a huge plus.
• These emails are sometimes time-sensitive (so can't take all day to send).

Thoughts? Tips? Point me in the right direction?

---

EDIT

To clarify:

I can do these on my own:

• maintain user list
• handle email content generation based on user preferences

And need something else (app, third-party service, w/e) to:

• accept email content and addresses and actually send the emails out
• provide tracking data (opens, clicks, etc). The more detail the better.

I'm leaning towards a third-party service, since I'm not sure any app can avoid choking the server when sending thousands of emails (though I wouldn't consider myself an email expert so I could be wrong).

Answer 1

Use Google AppEngine if you are worried about scalability & customization: it provides an email API and you can interface anything to it provided it is through an HTTP interface.

Of course, this is just a suggestion - disregard if this doesn't fit.

Answer 2

I usually got around this by having a mail "sending" function that dumped the emails into a queue (database table) with a job that ran every couple of minutes, grabbed the next x emails in the queue, sent those out and marked them as succeeded. That's the simple bones of it. You can then add on handling for email failures, returned mail, etc in version 2.

Answer 3

Why not keep your PHP system and use a third party SMTP service? Some reliable companies offer the use of e-mailing-only machines at reasonable prices, e.g. Dewahost who I am planning to use.

Also see the question Third Party Email Senders and my answer there.

Answer 4

Quite possibly not ideal, but if you're looking at large scale transmission there are commercial solutions such as Port 25's PowerMTA that can be set up to effectively transmit the contents of a given folder.

As such, you'd simply use PHP to create the personalised MIME formatted raw data for each outbound email and place these in a temporary directory prior to transmission. (I've written such a system in the past and you'd be surprised at the PHP's ability to grind out the emails, even with quite complex text & HTML emails with images as inline-attachments, etc.) Once you were ready to transmit, you'd then move the files en-masse to the PowerMTA monitored folder and it would take care of the transmission.

Depending on how you look at it the benefit/problem with such a solution is that you'll need to build trust relationships with people such as AOL, MSN/Hotmail, etc. to ensure that your mail server isn't blacklisted due to user's reporting email as SPAM. (That said, this will likely be a factor with any DIY solution.)

Answer 5

I'd recommend using the third party mailing service Silverpop, or something like it. We've used them for a few years and have been fairly satisfied. They already have relationships with the major email clients (AOL, Yahoo!, Gmail, etc.) and they do a good job of telling you if the stuff you're sending is likely to be classified as SPAM.

They have a fairly extensive API that uses XML HTTP/HTTPS requests that can tie in to existing systems. You can use it to remotely trigger emails, schedule mailings, customize email contents, set up, manage and query huge lists of recipients, run batch processes, etc.

It isn't a perfect service, but compared to a lot of others out there, they do pretty well. I have had very few complaints about them thus far.

Answer 6

Check out Campaign Enterprise for a possible in-house solution.

Answer 7

We use the built-in PHP email functionality to send thousands of emails to subscribers on a daily basis.

This is a terrible idea. It chokes out our server, and takes hours to complete the whole batch.

Why do you think that your problems are anything to do with the built-in PHP email function? It's a very thin wrapper around 'mail' or a simple SMTP client depending on what platform you are running on.

When you say it chokes your server - do you mean your email server? Your web server? something else?

There's nowhere near enough information here to make a proper diagnosis but it looks like the problems are of your own making - sure, there are lots of people out there who promise to sort all your problems for you if only you buy their latest product/service. But there's a very good chance that this isn't going to solve your current problems.

Can you tell us:

1. what OS the PHP is running on

2. how you invoke the code to create the emails

3. what the mail config in the php.ini file is

4. what type of MTA are you using? On what OS?

5. how is youe MTA copnfigured - does it deliver directly or via a smart relay?

6. which server is getting "choked"?

7. What anti-spam measures do you have in place for outgoing mail?

Then tell us what you've done to diagnose the fault and why you think its specifically on sending mails.

C.