Do you really need a study to work out that a Markov chain can’t solve problem questions? It feels pretty intuitive.
LLMs are not intelligent. Human intelligence is stored in text, so mimicking text mimicks that intelligence. The sheer quantity of text means they’ve probably seen something similar to the prompt.
If LLMs were an intelligent system they wouldn’t need to steal 1000TB of text and media content. Have you ever seen a person require a million books? It shows that the learning is brute force rather than intelligent.
It’s close, but it’s just not there. The methodology is wrong.
The Markov chain wouldn't fit in a googol of observable universes. The analogy is absolutely misleading. There's no reason to think that the chain of that humongous size cannot capture at least something of what humans with their limited brain size can do.
It cannot capture non in-context learning, true. But we have tools to modify the "Markov chain".
I guess your point is that it's obvious, but I'd argue it's not.
A lot of people are hyping up LLMs as the solution to all our problems, and there have been a lot of discussion around what's reasonable use cases (and a lot of anecdotal evidence trying to argue and support certain stances). My point is that we need (scientific) proof to move forward, and create confidence in "facts" we base our decisions on. Obviously that doesn't seem to be very fashionable nowadays - on both sides of the LLM discussion there seems to be a lot of people arguing hard for there stances. I'm very glad some are working on providing some facts into this discussion.
(one can argue about science vs truth, but regardless it's a step in the right direction/a good motive)
(this is btw why a solid educational foundation, not only skill focused, is very critical for society. cf democracy)
LLMs are not intelligent. Human intelligence is stored in text, so mimicking text mimicks that intelligence. The sheer quantity of text means they’ve probably seen something similar to the prompt.
If LLMs were an intelligent system they wouldn’t need to steal 1000TB of text and media content. Have you ever seen a person require a million books? It shows that the learning is brute force rather than intelligent.
It’s close, but it’s just not there. The methodology is wrong.