Hacker Newsnew | past | comments | ask | show | jobs | submitlogin

I'd be slightly more careful about making the claim you are making (though in your defense, nearly every person who makes a boastful post about their A/B testing is makes the same mistake).

For starters, you are picking an incredibly noisy metric, taking the maximum observed conversion and the minimum observed conversion and then dividing them. To better illustrate this, I wrote a little test script to simulate what you did many times. It assumes you have 4 email variations, send 125 emails each, and the average conversion rate is 5% across the buckets. Then it takes the minimum converting bucket, the maximum converting bucket, and divides them. Then does that 10k times and averages out what the gain would be by this metric, which is ~2.7x (code here: https://gist.github.com/3846278). 2.7x is a lot, but of course in this example it is pure noise, every single bucket converts exactly the same over the long term.

This metric also is particularly unhelpful because your goal is to have one high converting email. The metric you have chosen will be artificially boosted if you happen to have one horribly converting email. While that is potentially interesting fodder in just how differently various emails can convert, having one horrible email doesn't help you very much.

A better way for you to do this analysis is to dump your raw results into a mathematically sound A/B test calculator (I'm a bit of a homer but I don't think you can beat ABBA [http://www.thumbtack.com/labs/abba/]), then look at the confidence conversion ranges of the various emails and only make claims based on that. Like... I tried 4 versions of email copy and got my conversion rate up to X% (+/- some hopefully small confidence interval)! One of the emails was a real stinker and only converted at Y% (+/- confidence interval), thankfully I A/B tested first and didn't end up getting stuck with that one!



You make an important point. We obviously have a lot of testing and refining to do. Thank you for taking the time to investigate our claim :)




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: