It's a term of art from the days of image recognition AI that would confidently report seeing a giraffe while looking at a picture of an ambulance.

It doesn't feel right to me either, to use it in the context of generative AI, and I'd support renaming this behaviour in GenAI (text and images both) — though myself I'd call this behaviour "mis-remembering".

Edit: apparently some have suggested "delusion". That also works for me.