That is a fair point, but it would then apply to everything else we teach it about, like how we perceive the color of the sky or the taste of champagne. Should we remove these from the training set too?
Is it not still good to be exposed to the experiences of others, even if one cannot experience these things themself?
Thanks for saying it's a fair point, but it's more of an offhand joke about "an innocent machine". In reality, a machine, even an LLM, has no innocence. It's just a machine.
Having studied biology, I never accepted the "just a machine" argument. Everything is essentially a machine, but when a machine is sufficiently complex, it is rational to apply the Intentional Stance to it.
Is it not still good to be exposed to the experiences of others, even if one cannot experience these things themself?