Congrats Paras! New dashboard is looking nice, and I especially like the ability to retroactively segment results based on different customer dimensions and discover new opportunities for personalization.
One thing that I've noticed is that traditional A/B testing is a pretty sub-optimal way of answering the question: 'What works better, A or B?'
In the most basic example of an A/B test, you have a variation A and a variation B each shown to 50% of your user base. By definition, this approach will be sending half of your users to a worse performing version during the entire duration of the test!
The automated approach is based on a bandit algorithm that dynamically updates the proportion of users shown a given variation. With each new piece of data that you collect on the test variations' conversion rates and confidence, the algorithm adjusts the percentages automatically so that better performing variations are promoted and worse performers are pruned away.
This leads to:
1) faster results, because your directing test resources (i.e.: users and their data) to validate what you actually care about (i.e.: confidence in the best variation’s performance)
2) a higher average conversion rate during the test itself, because relatively more users are being sent to the better performing variation automatically, and
3) less time and effort required to actively manage your experiments.
Though the math behind this approach is slightly more complex than a traditional A/B test, it’s a no-brainer for those that are really interested in making data-driven decisions because of how much better the results are that it produces.
One thing that I've noticed is that traditional A/B testing is a pretty sub-optimal way of answering the question: 'What works better, A or B?'
In the most basic example of an A/B test, you have a variation A and a variation B each shown to 50% of your user base. By definition, this approach will be sending half of your users to a worse performing version during the entire duration of the test!
The automated approach is based on a bandit algorithm that dynamically updates the proportion of users shown a given variation. With each new piece of data that you collect on the test variations' conversion rates and confidence, the algorithm adjusts the percentages automatically so that better performing variations are promoted and worse performers are pruned away.
This leads to:
1) faster results, because your directing test resources (i.e.: users and their data) to validate what you actually care about (i.e.: confidence in the best variation’s performance)
2) a higher average conversion rate during the test itself, because relatively more users are being sent to the better performing variation automatically, and
3) less time and effort required to actively manage your experiments.
Though the math behind this approach is slightly more complex than a traditional A/B test, it’s a no-brainer for those that are really interested in making data-driven decisions because of how much better the results are that it produces.
For anyone interested, here’s a post we put together on how it works: http://splitforce.com/resources/auto-optimization/