Hacker Newsnew | past | comments | ask | show | jobs | submitlogin
How to A/B Test Using Javascript in Your Posterous Blog (rjmetrics.com)
19 points by rj_metrics on July 7, 2011 | hide | past | favorite | 3 comments


I say this all the time, but be very very careful split testing in JavaScript. It is best left for very small elements on your site. Otherwise your results will be misleading. Once you get to a nontrivial size implement it server side.


Can you elaborate on why you think the results will be misleading? Does it have to do with data integrity issues of some sort?


Because for large elements there is a delay, especially when your user has a slow browser and is talking to a third party service that is far away.

For example, say you ran a company that had about 100 paying users, almost all of which were in the SV tech scene. A fine opportunity to use JS because they all have fast browsers and the odds that the delay between KISS Metrics or Mixpanel and them are fairly low.

If, on the other hand, you sell Bingo Card Creators to thousands of teachers around the globe the latency is going to kill you. A teacher in England will see the button (s)he wants to click on, then all of a sudden it is gone and replaced with a intro video, or, even worse, the prices have all gone up and there are different options on the pricing page.

Not only is your split test reporting the wrong winner (because of a negative bias on the B side of the test) but it can also erode the confidence visitors had in your service.




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: