I've always thought that these types of studies / meta-analysis are conducted in a backward manner. A study is proposed, funded, performed - possibly with a particular objective in mind. The results are published (possibly used in various marketing materials), then almost enviably, methods and rationale are questioned by others post hoc. The end result of the exercise is often no knowledge gained with diminishing public trust in science.
Instead what we should do, is to come up with a study design first, gain broad consensus that the study, if rigorously conducted in accordance to the design, would yield statistically meaningful results.
The challenge here is many studies would likely be infeasible (requiring millions of people or hundreds of years in order to attain a statistically valid result - or both). However, that is useful in it's own right as it would help us to direct resources at studies that do have a hope of increasing knowledge. Infeasible studies will just have to wait until we can accurately simulate biological phenomena.
Instead what we should do, is to come up with a study design first, gain broad consensus that the study, if rigorously conducted in accordance to the design, would yield statistically meaningful results.
The challenge here is many studies would likely be infeasible (requiring millions of people or hundreds of years in order to attain a statistically valid result - or both). However, that is useful in it's own right as it would help us to direct resources at studies that do have a hope of increasing knowledge. Infeasible studies will just have to wait until we can accurately simulate biological phenomena.