What's the best way to A/B test Email Subject Lines in Rails?

Question

I'm sending out a lot of emails in my latest rails app. I want to A/B test subject lines in the emails that go out. ideally i can capture two things.

1- open rate 2- whether the call to action in the email is clicked

any ideas on how to do this? i don't think (the great) 7 minute abs (http://github.com/paulmars/seven_minute_abs/tree/master) will do this because @subject is set in the model, while the ab test param for the querystring is assigned in the view, and the test versions will likely be different.

thanks!

Answer 1

Could you sendout a custom signup link that had a querystring on the end e.g. example.com/signup?one or example.com/signup?two then check for that query string at signup?

I know this isn't Ruby on Rails but it might be worth looking at and seeing how they plan to do it.

http://www.campaignmonitor.com/blog/post/2782/ab-testing/

Answer 2

I've been looking at this as well. 7 minute abs is a nice A/B solution, but I think for emails you can do it yourself relatively easily. If you have either a dedicated image (say an invisible "bug" image) that you use for open detection, or just an image that is always included, you can put a param on that, or rather, just alias it in your web server, or not alias it for that matter, and just copy the image, e.g. "open_a.gif" and "open_b.gif" and watch your Analytics or web logs to track which gets opened more. If you don't need to associate it with the email recipient, then you can simply send half the emails using the A image, and half with B, etc. Same goes for the URL in the email for case #2 as you mention, that one can probably be a simple URL parameter or path, and you just let Analytics track that for you.

The Campaign Monitor solution, and Campaign Monitor in general, is excellent, if it can work for you. We use Campaign Monitor for our newsletter emails, but can't for all the personalized email we send. If you are sending the same email to all users, then by all means, I'd go with Campaign Monitor, but if each mail is personalized you'll probably need to roll your own.