It is clear to me that humans “hallucinate” all the time, and I don’t see why this should disqualify AI. One prominent human hallucinated that a hurricane was going to go in a particular direction and kindly updated a map, provided by scientists, with a sharpie.
Aaand you’re saying humans don’t do that? There are plenty of humans who will just make shit up with whatever information they have. There’s plenty of humans who wouldn’t even know the British don’t use dollars and would just put a dollar sign there.
I’ve seen a senior engineer at a FAANG confidently interrupt another engineer, who was giving the correct answer, to provide an incorrect answer. They mixed up facts and confidently stated a conclusion that sounded good, but was utterly wrong.