Also, doesn't their suggested approach amount to multiple testing? In other word... | Hacker News

Hacker Newsnew | past | comments | ask | show | jobs | submit

		hnhg on June 16, 2023 \| parent \| context \| favorite \| on: Annoying A/B testing mistakes Also, doesn't their suggested approach amount to multiple testing? In other words, a kind of p-hacking: https://en.wikipedia.org/wiki/Multiple_comparisons_problem Edit - and this: http://www.stat.columbia.edu/~gelman/research/unpublished/p_...

throwaway202351 on June 16, 2023 [–]

Yeah, a good AB testing framework would either refuse to let you break things down too much or have a large warning about the results not being significant, but that doesn't always stop the business-types from trying to wiggle in some way for them to show a win.

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact