Unit Testing Email Content

Question

I'm usually one to say any piece of code no matter what it does, especially if it's runs in a production environment should be tested. I've been wondering, however, if dynamic email content should be an exception. It changes so often and is usually a pain to properly and fully test that I don't know if it's worth it. More often than not, the content of the unit test is copy/pasted anyway (I know this is not ideal, but happens nonetheless), so it's not really giving any real benefit other than telling us if something will severely blow up, and a simple unit test just trying to send the email (not verifying the copy) should take care of that.

I was hoping to get some opinions from other developers. Let me know what you think. Unit test email content or not?

Edit: Just to clarify, we already have separate unit/integration tests for the actual sending of emails, this question is referring to just testing the content of an email. Currently we unit test only the dynamic content, the template text is not aside from click testing.

Answer 1

I would add a unit test that verifies that content can be placed in the email from a template (or other dynamic source). That content can be specific to this unit test like "This is a test email to ##USERNAME##" or similar. If the template changes, I would not update your unit test with new template information.

Answer 2

My basic rule is this: if it affects the functioning of the application, test it. (Yes, that's a pretty loosely-defined rule.)

As such, I'd note that a lot of the e-mails that sites send are things like user activations. In these cases, I wouldn't test the content of the e-mail per se (that's content, not function), but I would verify that it was sending out the correct activation URL.

I'd extend that to any piece of dynamic e-mail content that is generated by code, from data that isn't a hard-coded string message.

Answer 3

When it comes to testing systems similar to the one you suggest, my view is that you are testing a full system rather than a logical block of code. I would write unit tests to test the smaller units of code within the system and some of the connected parts and test the complete system using functional testing with a user.

Answer 4

You should always test everything, yes, but you can't always unit test everything. In this case, trying to verify that an email template has been produced correctly is probably better off in integration and user acceptance testing.

You can unit test how your programs build email templates, however, and this is the route I recommend you go.

Build an API for filling an email template, a helper class with methods like:

// using C# syntax returning strings for the example -- you could just as easily return
// System.Net.Mail.MailMessage or javax.mail.Message instead
string BuildPasswordChangeTemplate(string username, string newPassword, string email);
string BuildErrorTeplate(string methodName, string serviceName, Exception e);

As long as the methods are defined in an interface or virtual (and aren't static), you can mock the helper class and unit test that your code calls into the appropriate template builder at the appropriate time. You can then push spelling and format and those other things to user acceptance testing and consider your work done.

Answer 5

I see two distinct things you would want to test here. Testing the generated e-mail content is one, and you should be able to refactor your code so that you can unit test just that part. You can write a function or class that handles only the generation of the e-mail, and then write unit tests around that for the different inputs you're interested in.

If the e-mail content is part of your code, you should have some kind of test around it. But maybe you want more of a sanity check than verifying that the e-mail content exactly matches "Congratulations {NAME}, you've just won the Nigerian lottery...". Perhaps just check that the content is over a certain size threshold and that it contains the recipient's name (or whatever dynamic content gets inserted) somewhere in the body?

The second thing is testing the mail sending machinery. That's not strictly a unit test; I consider that a functional or integration test. If you have a QA team or process that handles this type of large scale testing already, you can probably safely punt on this. If not, it's not hard to write a small stub SMTP server to accept incoming mail and run that as part of your functional test suite. SMTP is a pretty simple protocol, and you only need to implement a half dozen or so commands to accept mail messages. I wrote one a while ago using Ruby in about a day. You'll need to be able to reconfigure the SMTP host and port your app uses so you can set one for test and a different one for production.

Answer 6

Unit tests to ensure that your all your merge codes work if it's a finite set, or the merge mechanism works if it is dynamic. As well as a test that template text makes it to the merged document makes sense.

After that testing each template shouldn't really add any value, but does add a bunch of work.