Emails sometimes get scrambled

Question

Folks,

I have a PHP-based site (using the QCubed framework); as a part of the site, I have a daemon that's sending out several thousand emails a day (no i'm not a spammer, everything is opt-in :)). Emails are sent through a custom framework component; that component serves as an SMTP client. I'm using a paid SMTP gateway from DNSExit.com to get the emails actually delivered.

Those emails are simple HTML-based emails; they really have just simple links inside.

My issue is that these links sometimes (not consistently!) get scrambled during transition. Tags somehow get mixed up, and some links are non-functional in the email. The issue happens on a small percentage of all sent emails; it is not consistent (i.e. the same exact source message HTML may or may not cause the scrambling in transition).

Have any of you seen this? Any thoughts on how to troubleshoot?

Answer 1

Had 2 problems with email data - usually "?" symbol somehow got inside some words, another was UTF and title related. First got "fixed" by changing hosting provider (so it was mail-server related) second one got fixed by changing PHPmailer library.

Try to specify how exactly data is scrambled.

Answer 2

Is it possible that you are using temp files to create the emails (or at minimum to create the variable content)? I did something vaguely similar once upon a time. The email text was generated and written to a temp file based on the exact time in seconds. Unfortunately, when sending thousands per day, we were hitting the same second more than once (since there are only 86k seconds available). That might explain a) the small error rate and b) the apparent randomness. For troubleshooting, I'd just see if the error rate increases with the number of emails and go from there.

Answer 3

Have you any special attributes in your links? May be title attribute with not escaped quotes inside?

Something like this: <a href="http://some.site" title="My "correct" link">Link</a>

Answer 4

I ran into a similar problem on a server running sendmail.

I was creating and testing an html email that would one day be mass mailed (opt-in, of course). I had myself a template for the email that was easy for any html programmer to read, but as such was heavy on the whitespace to line everything up correctly. I thought to myself, if this is going to be mass emailed, after the template is rendered, I think I will minimize the whitespace in the file to save on space! So I created a brilliant regular expression to rid any unnecessary to send whitespace from the rendered email.

Upon sending the email to myself, I opened the email and was baffled when I saw that some of the css and html were not showing up correctly, when my previous emails prior to my regexp were. By looking at the original message I noticed that every once in a while, an exclamation mark (!) was appearing seemingly randomly throughout the message, thus breaking any css and html that came in its random path.

Turns out that sendmail doesn't like it if a line in your email gets too long without a line break. When the line does get too long, sendmail will insert an exclamation mark followed by a line break right then and there, just to confuse and confound you.

Why did it not just choose a space between words to line break? Why insert the exclamation mark? Questions I'm afraid, without answers.

My solution?

sudo apt-get remove sendmail
sudo apt-get install exim4

I was having other problems with sendmail like it taking a full 60 seconds to send an email and exim4 just worked and I have never had to think about it again.

If your mail server is using sendmail, this very well could be the problem, if not, thank you for letting me share my story with you. I needed to vent.

Answer 5

When you're sending email you should encode it so every line in the message body is not longer then 76 characters. You could use base64 for this but most systems use the quoted-printable encoding for text because it generates smaller messages. Base64 is usually only used for binary data.

Answer 6

The problem is that HTML is not compatible with email. That is why I created Mail Markup Language.

HTML was created to operate with the HTTP protocol as those two technologies were invented by the same person at about the same time. The difference is that HTTP is a single session one way transfer from a server to a client. That never changes as the HTML document always originates on a server, is sent to a requesting client, and once the transfer completes the connection between the client and server is dropped.

Email does not behave in such a way. In email a communication originates at a client, is sent to one or more email serves, and then terminates at a distant client. The biggest difference, however, is that the document does not die with finality of a single transmission instance as is the case with a document transfer over HTTP. A document sent in SMTP can be replied to, forwarded, or copied to multiple unrequested users. This one difference is profound when consideration for an email thread is considered.

The problem is that SMTP and HTTP are different as demonstrated in the prior two paragraphs. This differences is compounded in that SMTP and HTTP have radically different formatting methods for the creation of header data. HTML has header data that is intended to be compatible with the headers of HTTP transmissions and offer no compliance to SMTP transmissions. The HTML headers also do not account for the complexity of an email thread.

The problem is exemplified when email software corrupts a HTML document to add formatting changes necessary to fit the conforming demands of that software and to also write header data directly into the document. This exemplification becomes extremely pronounced when an HTML email becomes an email thread. Since the HTML header data has no method to account for the complexities of an email thread there is no way to supply relevant presentation definitions from a stylesheet that survive the transfer of the document. Each time a HTML document, or a document with HTML formatting, is sent from one email software to another the document is corrupted and each email software device corrupts the prior corruption. Email processing software may refer to either an email client, which certainly will corrupt a document, or an email server, that may only likely corrupt an email document.

The solution to the problem is to create a markup language convention that recognizes the requirements of email header data directly. Those requirements are defined in RFC 5321 for the SMTP protocol and RFC 5322 for the client processing. The only way to properly extend this solution to account for the complexities of an email thread are to provide a convention for a multi-agent DOM.

Paragraphs deleted due to technical inaccuracy and difference between the term multi-agent DOM and the nature of an invented feature not mentioned here even prior to the edit.

EDIT: a multi-agent DOM applies some degree of hierarchy, which may not be necessary to represent an email thread.