Decide your sample size beforehand, and only stop the test after you've reached it.
That may be the gospel among A/B testers, but from a Bayesian perspective, this is definitely not the best you can do (though it won't necessarily be wrong). Beware, Bayesian methods are provably optimal.
The details are here for the interested (chap. 4 & 6):
Thank you, I downloaded the Bayesian method paper linked in the article and was going to read it, I'm very interested in a probably optimal stopping method (obviously). I'm only afraid it'll fly over my head, despite my having a bit of stats background.
Berry, Donald A. “Bayesian Statistics and the Efficiency and Ethics of Clinical Trials,” Statistical Science, Vol. 19, No. 1 (Feb., 2004), pp. 175-187
I haven't heard of it. But if you're seriously motivated to do things right and you're "mathematically mature", I would advise you to read Jaynes' book linked above. For me it was a huge eye opener. The screed of the Bayesians is "You don't need a separate statistics field! You just need the laws of probabilities, and that's enough to answer any question you want."
Let me know if you need help, my email is in my profile.
StavrosK, I'm delighted to see you post this; I was hoping you'd do so!
If people are interested in calculating the sample size before running an A/B test, I'd recommend tools such as the following (to which I have no connection):
That may be the gospel among A/B testers, but from a Bayesian perspective, this is definitely not the best you can do (though it won't necessarily be wrong). Beware, Bayesian methods are provably optimal.
The details are here for the interested (chap. 4 & 6):
http://www-biba.inrialpes.fr/Jaynes/prob.html
It's not casual reading... I wish there was something shorter available.