>> Technically, I did find two significant results.
The problem is that he is comparing two very different things, the level when he took the pill and the level one hour after that. So it's not surprising that they are different. Let's imagine a very very very stupid experiment, where the problem is more obvious.
Does Coca Cola or Pepsi improve luck? N=1000000, double blind randomized controlled trial.
1) Each subject flips a coin. tail=0, head=1.
2) They drink a glas of soda, 50% Coke or 50% Pepsi, that is served in a hidden place and nor the subject or the experimenter know which one.
3) They roll a dice (an usual one, D6)
Results:
* Average before Coke = 0.5002
* Average after Coke = 3.5005
* Average before Pepsi = 0.5004
* Average after Pepsi = 3.5003
So the conclusion is that Coke improves the average (p<1E-a-lot) and Pepsi improves the average (p<1E-a-lot). Both are "technically" statistically significant (but it's caused by a horrible experiment design).
Unsurprisingly, the difference in the average after drinking Coke or Pepsi is not statistically significant (p<.something).
(I'm too lazy to run a simulation now, but it's not difficult to get realistic averages and p values.)
In conclusion, the useful result is the comparison of the anxiety after taking both drugs, not the difference of before and after taking them.
The problem is that he is comparing two very different things, the level when he took the pill and the level one hour after that. So it's not surprising that they are different. Let's imagine a very very very stupid experiment, where the problem is more obvious.
Does Coca Cola or Pepsi improve luck? N=1000000, double blind randomized controlled trial.
1) Each subject flips a coin. tail=0, head=1.
2) They drink a glas of soda, 50% Coke or 50% Pepsi, that is served in a hidden place and nor the subject or the experimenter know which one.
3) They roll a dice (an usual one, D6)
Results:
* Average before Coke = 0.5002
* Average after Coke = 3.5005
* Average before Pepsi = 0.5004
* Average after Pepsi = 3.5003
So the conclusion is that Coke improves the average (p<1E-a-lot) and Pepsi improves the average (p<1E-a-lot). Both are "technically" statistically significant (but it's caused by a horrible experiment design).
Unsurprisingly, the difference in the average after drinking Coke or Pepsi is not statistically significant (p<.something).
(I'm too lazy to run a simulation now, but it's not difficult to get realistic averages and p values.)
In conclusion, the useful result is the comparison of the anxiety after taking both drugs, not the difference of before and after taking them.
As the article says:
>> So I propose a new rule: Blind trial or GTFO.