I agree with this sentiment and would remind everyone that LLMs are probabilistic models. Anything that isn't in the training data set will not be produced as an output. If you squint really hard, you could kinda say an LLM is just a fancy compression/decompression scheme.
That said, in addition to anthropomorphizing something that sounds like a person, I think a fascinating thing about these discussions are the variety of unstated opinions on what reasoning, thought, or awareness is. This whole topic would be a lot simpler to classify if we had a better understanding of how _we_ are able to reason, think, and be aware to the level of understanding we have of the math behind an LLM.
> Anything that isn't in the training data set will not be produced as an output.
Thinking back to some of the examples I have seen, it feels as though this is not so. In particular, I'm thinking of a series where ChatGPT was prompted to invent a toy language with a simple grammar and then translate sentences between that language and English [1]. It seems implausible that the outputs produced here were in the training data set.
Furthermore, given that LLMs are probabilistic models and produce their output stochastically, it does not seem surprising that they might produce output not in their training data.
I agree that we do not have a good understanding of what reasoning, thought, or awareness is. One of the questions I have been pondering lately is whether, when a possible solution or answer to some question pops into my mind, it is the result of an unconscious LLM-like process, but that can't be all of it; for one thing, I - unlike LLMs - have a limited ability to assess my ideas for plausibility without being prompted to do so.
> remind everyone that LLMs are probabilistic models
The problem is that's overly reductive. You and I are also probabilistic.
That's not to say an AI must therefore be like us, as the entire universe and everything in it is also a probabilistic thanks to quantum mechanics.
Every proposed test of "real intelligence" I've seen in these threads has much the same issue, it's just that some of the tests count too many things as intelligent (e.g. a VHS tape and recorder can "experience input", record that as a "memory", and "recall" it later) or excludes all humans (e.g. by requiring the ability to violate Gödel's incompleteness theorem, a standard which surprisingly even comes up on this site).
I'm happy to describe even the most competent LLM as "dumb" in the IQ sense, even though I also say they're book-smart — because even last year's models were a demonstration proof that a brain the size of a rat's can, with 2.5 million years of subjective experience, reach the performance of a work placement student in almost every book oriented field at the same time.
That said, in addition to anthropomorphizing something that sounds like a person, I think a fascinating thing about these discussions are the variety of unstated opinions on what reasoning, thought, or awareness is. This whole topic would be a lot simpler to classify if we had a better understanding of how _we_ are able to reason, think, and be aware to the level of understanding we have of the math behind an LLM.