Best practice for assigning A/B test variation based on IP address

Question

I am starting to write some code for A/B testing in a Grails web application. I want to ensure that requests from the same IP address always see the same variation. Rather than store a map of IP->variant, is it OK to simply turn the IP address into an integer by removing the dots, then use that as the seed for a random number generator? The following is taking place in a Grails Filter:

def ip = request.remoteAddr
def random = new Random(ip.replaceAll(/\./, '').toInteger())
def value = random.nextBoolean()
session.assignment = value
// value should always be the same for a given IP address

I know that identifying users by IP address is not reliable, and I will be using session variables/cookies as well, but this seems to be useful for the case where we have a new session, and no cookies set (or the user has cookies disabled).

Answer 1

You could simply take the 32-bit number and do ip mod number_of_test_scenarios. Or use a standard hashing function provided in ruby. But I feel I should point out a few problems with this approach:

1. If your app is behind any proxies, the ip will be the same for all the users of that proxy.
2. Some users will change IPs fairly frequently, more frequently than you think. Maybe (as Joel Spolsky says) "The internet is broken for those users", but I'd say it's a disservice to your customers if you make the internet MORE broken for them, especially in a subtle way, given that they are probably not in a position to do anything about it.
3. For users who have a new session, you can just assign the cookie on the first request and keep the assignments in memory; unless a user's initial requests go to multiple servers at the same time this should resolve that problem (it's what I do on the app I maintain).
4. For users with cookies disabled, I'd say "The Internet is broken", and I wouldn't go to much trouble to support that case; they'd get assigned to a default test bucket and all go there. If you plan to support many such users in a non-broken way you're creating work for yourself, but maybe that's ok. In this case you may want to consider using URL-rewriting and 302 redirects to send these users down one scenario or another. However in my opinion this isn't worth the time.
5. If your users can log into the site make sure you record the scenario assignments in your database and reconcile the cookie/db discrepancies accordingly.