A/B testing and stats solutions

Question

I've been looking for a good testing framework for months, not finding anything, so I've just been building my own.

This is what I want to do:
- track arbitrary behaviors (e.g. # of photos viewed, # of comments posted)
- track correlation between arbitrary variables and those behaviors
(e.g, how do different versions of this prompt affect average # of photos viewed?)

This kind of thing should be a core part of agile development. What's out there? I know Google Website Optimizer is one of the answers, but you can only track behaviors that end in a single "success" page.

It'd be great to have a plugin that can work within your code (Rails in my case) and feed into a nice hosted service with pretty graphs...

Answer 1

You may want to partition your problem into analytics (impressions, actions, and possible in-page events, reporting of your tests), and a framework for serving up your variations (how do you manage x variations both in practical terms of preparing them, do you need to store variations for future reference, turning on and off tests, optimize the effectiveness of your test etc). There is clearly an overlap, say, Google Website Optimizer can turn off a bad variation as soon as it has data to support it, but by thinking about this as different problems you may be able to reuse perhaps the Google Analytics component.

Answer 2

Yep, here:

http://github.com/paulmars/seven_minute_abs/tree/master