I don't think it's accurate anymore to say LLMs are just really good word predictors. Especially in the last year, they are trained with reinforcement learning to solve specific problems. They are functions that predict next tokens, but the function they are trained to approximate doesn't have to be just plain internet text.
Yeah, that's fair. It's probably more accurate to call them sequence predictors or general data predictors than to limit it to words (unless we mean words in the broad, mathematical sense) they are free monoid emulators
The parent comments were attempting to characterize LLMs as something more general than “word predictors”. The alternative “sequence predictors” was proposed.
My question relates to whether we have any reason to believe that the relevant aspects of human cognition are anything more than that.
Certainly humans have some advantages, like the ability to continuously learn (although there’s very strong evidence that we have a pretraining phase too, for example the difficulty of learning new languages as an adult vs. as a child.) But fundamentally, it’s not clear to me that our own language production skills aren’t “just” sequence prediction.
Perhaps, as the OP article speculates, there are other important components, like “models of the world”. But in that case, it may be that we’re augmented sequence predictors.
