The need for representative sampling is something understood by first year undergraduates on first exposure to the concept.
What you've written is the reddit equivalent of criticizing a study with "correlation does not imply causation".
Yeah, scientists are familiar with the idea, believe it or not. Scientists in fact go to great lengths to ensure their sampling is representative, and studies published in good journals are carefully scrutinized for non-representative sampling.
If you can't conceive of how a person working at Mechanical Turk is overwhelmingly likely to just be a normal person, and not your weird conception of them, I'd encourage you to read Thinking Fast and Slow.
Even if you grant the idea that a person working at MTurk is just a normal person, which personally I doubt. You have to contend with both the fact that they're highly likely to be lying about their identity in order to maximize the number of tasks they can be included in, and they're also likely giving trash answers in order to complete the survey as fast as possible. I spent a bit of time doing mturk for a while, and it was absolutely one of the first things I started to optimize - "How do I get through this survey quicker". It's incredibly difficult to earn minimum wage doing mturk, which brings us back to the idea that it's unlikely it's just an average normal person doing it.
It's absolutely possible to produce a rigorous statistical result from the data, but that's useless if the data is all lies.
> Even if you grant the idea that a person working at MTurk is just a normal person,
Just want to point out that surveys are not looking for "normal" participants. They need a representative sample which, given human diversity, would include many people who are "abnormal" across a wide array variables.
Understanding the need for representative samples is not the same as actually getting representative samples. It's not a problem of scientists being ignorant or lazy or even underfunded. They know what they need, and they simply cannot get it. They try to make do anyway, but the quality of their work still suffers because of these factors which are out of their control.
Not all fields of inquiry are equally tractable to the scientific method, so they should not be treated with equal reverence.
I think his point is that it's impossible to normalise such a sample. For example if I sampled only toddlers about their attitudes to ice cream then there's no way to extrapolate from that sample to the general population, and it's the same thing for "people who are willing to fill in a survey for $2".
Without getting into snark, your very own article bolsters my point:
> The replication crisis in psychology does not extend to every line of inquiry, and just a portion of the work described in Thinking, Fast and Slow has been cast in shadows. Kahneman and Tversky’s own research, for example, turns out to be resilient. Large-scale efforts to recreate their classic findings have so far been successful. One bias they discovered—people’s tendency to overvalue the first piece of information that they get, in what is known as the “anchoring effect”—not only passed a replication test, but turned out to be much stronger than Kahneman and Tversky thought.
What portion of the scientists in the book are represented by Kahneman and Tversky? The other scientists in the book did not go to great lengths to avoid sampling bias. The authors could not be relied on to spot it either.
I don't see any point you made being bolstered. The anchoring effect you mention in another comment doesn't permit a scientist or statistician to throw their hands up in the air and say one mechanical turk group is as average as any other pseudo random group.
Thanks. I had the impression this book and his work was widely regarded. I'll have to put a mental asterisk next to some of the stories I was blown away by.
The book not so much (despite its initial critical acclaim on continuing commercial success), but at least his research summarized within seems to be reproducible and his reputation is intact. The research on priming in particular, was reliant on low sample size studies, and was not reproducible.
Do you have any data or evidence to support your claim that workers on Mechanical Turk are a representative sample of Americans? This is an extraordinary claim, given the incentives of the platform. Think like a hacker, or a poor person in India or Nigeria.
What you've written is the reddit equivalent of criticizing a study with "correlation does not imply causation".
Yeah, scientists are familiar with the idea, believe it or not. Scientists in fact go to great lengths to ensure their sampling is representative, and studies published in good journals are carefully scrutinized for non-representative sampling.
If you can't conceive of how a person working at Mechanical Turk is overwhelmingly likely to just be a normal person, and not your weird conception of them, I'd encourage you to read Thinking Fast and Slow.
Contempt for scientists is not a good look.