They get it wrong because no one knows how to do it right. Actually...evaluating human knowledge and behavior into a bunch of 1 hour interview slots doesn't lead to much accurate information.
They have the same problem as academics do, only without the benefit of being able to consistently apply the scientific method and good experimental practices (the only hope of cracking that nut).