That's exactly it. The bot espouses facts with the same tone of confidence regardless of whether they're true or entirely fictional.
I understand it has no sense of knowledge-of-knowledge, so (apparently) no ability to determine how confident it ought to be about what it's saying — it never qualifies with "I'm not entirely sure about this, but..."
I think this is something that needs to be worked in ASAP. It's a fundamental aspect of how people actually interact. Establishing oneself as factually reliable is fundamental for communication and social cohesion, so we're constantly hedging what we say in various ways to signify our confidence in its truthfulness. The absence of those qualifiers in otherwise human-seeming and authoritative-sounding communication is a recipe for trouble.
This is a particular alignment issue. People are used to people spouting bullshit all the time, as long as it's aligned to what we are used to. Take religion for example. People tend to be very confident around the unknowable there.
It is scary in the sense that people love following confident sounding authoritarians, so maybe AI will be our next world leader.
I understand it has no sense of knowledge-of-knowledge, so (apparently) no ability to determine how confident it ought to be about what it's saying — it never qualifies with "I'm not entirely sure about this, but..."
I think this is something that needs to be worked in ASAP. It's a fundamental aspect of how people actually interact. Establishing oneself as factually reliable is fundamental for communication and social cohesion, so we're constantly hedging what we say in various ways to signify our confidence in its truthfulness. The absence of those qualifiers in otherwise human-seeming and authoritative-sounding communication is a recipe for trouble.