The interesting problem is that if you set a strict significance threshold but have a sample size that is too small, you will still sometimes correctly get significant results, but the effect sizes will all be exaggerated.
If the sample size is too low, the only effect size big enough to be significant may be one that is much larger than the truth. So you only claim significance when you get lucky and exaggerate.
This is actually an enormous problem, and it probably affects physics too. Many medical and biological papers tout huge effects which turn out to be completely wrong -- not because their result was a false positive but because it's an exaggerated true positive. Sample sizes tend to be hilariously inadequate in soft sciences where each data point costs thousands of dollars.
I call this "truth inflation," although I don't know if it's been discussed enough to have a common name. It's heavily discussed in my book, if you're interested in seeing why the backlash against p values is so widespread: http://www.statisticsdonewrong.com/regression.html#truth-inf...
If the sample size is too low, the only effect size big enough to be significant may be one that is much larger than the truth. So you only claim significance when you get lucky and exaggerate.
This is actually an enormous problem, and it probably affects physics too. Many medical and biological papers tout huge effects which turn out to be completely wrong -- not because their result was a false positive but because it's an exaggerated true positive. Sample sizes tend to be hilariously inadequate in soft sciences where each data point costs thousands of dollars.
I call this "truth inflation," although I don't know if it's been discussed enough to have a common name. It's heavily discussed in my book, if you're interested in seeing why the backlash against p values is so widespread: http://www.statisticsdonewrong.com/regression.html#truth-inf...