So the thing is, giving wrong answers with confidence is literally what we train students to do when they are unsure.
I can remember my GRE coach telling me that it was better to confidently choose an answer I only had 50% confidence in, rather than punt on the entire question.
AIs hallucinate because, statistically, it is 'rewarding' for them to do so. (In RLHF)
I can remember my GRE coach telling me that it was better to confidently choose an answer I only had 50% confidence in, rather than punt on the entire question.
AIs hallucinate because, statistically, it is 'rewarding' for them to do so. (In RLHF)