Everyone is out here acting like "predicting the next thing" is somehow fundamentally irrelevant to "human thinking" and it is simply not the case.
What does it mean to say that we humans act with intent? It means that we have some expectation or prediction about how our actions will effect the next thing, and choose our actions based on how much we like that effect. The ability to predict is fundamental to our ability to act intentionally.
So in my mind: even if you grant all the AI-naysayer's complaints about how LLMs aren't "actually" thinking, you can still believe that they will end up being a component in a system which actually "does" think.
Are you a stream of words or are your words the “simplistic” projection of your abstract thoughts? I don’t at all discount the importance of language in so many things, but the question that matters is whether statistical models of language can ever “learn” abstract thought, or become part of a system which uses them as a tool.
My personal assessment is that LLMs can do neither.
Words are the "simplistic" projection of an LLM's abstract thoughts.
An LLM has: words in its input plane, words in its output plane, and A LOT of cross-linked internals between the two.
Those internals aren't "words" at all - and it's where most of the "action" happens. It's how LLMs can do things like translate from language to language, or recall knowledge they only encountered in English in the training data while speaking German.
LLMs are compression and prediction. The most efficient way to (lossfully) compress most things is by actually understanding them. Not saying LLMs are doing a good job of that, but that is the fundamental mechanism here.
It's the other way around. Human learning would appear to amount to very efficient compression. A world model would appear to be a particular sort of highly compressed data set that has particular properties.
This is a case where it's going to be next to impossible to provide proof that no counterexamples exist. Conversely, if what I've written there is wrong then a single counterexample will likely suffice to blow the entire thing out of the water.
No answer I give will be satisfying to you until I could come up with a rigorous mathematical definition of understanding, which is de-facto solving the hard AI problem. So there's not really point in talking about it is there?
If you're interested in why compression is like understanding in many ways, I'd suggest reading through the wikipedia article on Kolmogorov complexity.
The "cross-linked internals" only go one direction and only one token at a time, slide window and repeat. The RL layer then picks which few sequences of words are best based on human feedback in a single step. Even "thinking" is just doing this in a loop with a "think" token. It is such a ridiculously simplistic model that it is vastly closer to an adder than a human brain.
My "abstract thoughts" are a stream of words too, they just don't get sounded out.
Tbf I'd rather they weren't there in the first place.
But bodies which refuse to harbor an "interiority" are fast-tracked to destruction because they can't suf^W^W^W be productive.
Funny movie scene from somewhere. The sergeant is drilling the troops: "You, private! What do you live for!", and expects an answer along the lines of dying for one's nation or some shit. Instead, the soldier replies: "Well, to see what happens next!"
I doubt words are involved when we e.g. solve a mathematical problem.
To me, solving problems happens in a logico/aesthetical space which may be the same as when you are intellectually affected by a work of art. I don't remember myself being able to translate directly into words what I feel for a great movie or piece of music, even if in the late I can translate this "complex mental entity" into words, exactly like I can tell to someone how we need to change the architecture of a program in order to solve something after having looked up and right for a few seconds.
It seems to me that we have an inner system that is much faster than language, that creates entities that can then beslowly and sometimes painfully translated to language.
I do note that I'm not sure about any of the previous statements though'
My wordmangling and mathsolving happen in that sort of logico/aesthetical space, too!
The twist about words in particular is they are distinctly articulable symbols, i.e. you can sound 'em out - and thus, presumably, have a reasonable expectation for bearers of the same language to comprehend if not what you meant then at least some vaguely predictable meaning-cloud associated with the given speech act.
That's unlike e.g. the numbers (which are more compressed, and thus easier to get wrong), or the syntagms of a programming language (which don't even have a canonical sonic representation).
Therefore, it's usually words that are taught to a mind during the formative stages of its emergence. That is, the words that you are taught, your means of inner reflection, are still sort of an imposition from the outside.
Just consider what you life trajectory would've been if in your childhood you had refused to learn any words, or learned them and then refused to mistake them for the things they represent!
Infants and even some animals recognize their reflection in a mirror; however, practically speaking, introspection is something that one needs to be taught: after recognizing your reflection you still need to be instructed what is to be done about it.
Unfortunately, introspection needing to be taught means that introspection can be taught wrongly.
As you can see with the archetypical case of "old and wise person does something completely stupid in response to communication via digital device", a common failure mode of how people are taught introspection (and, I figure, an intentional one!) is not being able to tell apart yourself from your self, i.e. not having an intuitive sense of where the boundary lies between perception and cognition, i.e. going through life without ever learning the difference between the "you" and the "words about you".
It's extremely common, and IMO an extremely factory-farming kind of tragic.
I say it must be extremely intentional as well, because the well-known practice of using "introspection modulators" to establish some sort of perceptual point of reference (such as where the interior logicoaeshtetical space ends and exterior causalityspace begins) very often ends up with the user in, well, a cage of some sort.
What is your point exactly in regard to what I said earlier, how would you rephrase what you just said as a philosophical/epistemological statement ?
> It's extremely common
I cannot conceive this ? I am lacking the empirical knowledge you seem to have. (I don't understand your "archetypical case", I can't relate to it). I'd love a reexplanation of your point here, as your intent is unclear to me.
I didn't understand also the "introspection modulators" part :(, (a well known practice ?? I must be living on another planet haha...).
edit: or maybe that's a metaphor for "language" ??
Different people have different levels of internal monologuing or none at all. I don't generally think with words in sentences in my head, but many people I know do.
Internal monologue is a like a war correspondent's report of the daily battle. The journalist didn't plan or fight the battle, they just provided an after-the-fact description. Likewise the brain's thinking--a highly parallelized process involving billions of neurons--is not done with words.
Play a little game of "what word will I think of next?" ... just let it happen. Those word choices are fed to the monologue, they aren't a product of it.
Most of the fucking time, and I would prefer that I didn't. I even wrote that, lol.
I don't think they're really "mine", either. It's just all the stuff I heard somewhere, coalescing into potential verbalizations in response to perceiving my surroundings or introspecting my memory.
If you are a materialist positivist, well sure, the process underlying all that is some bunch of neural activation patterns or whatever; the words remain the qualia in which that process is available to my perception.
It's all cuz I grew up in a cargo cult - where not presenting the correct passwords would result in denial of sustenance, shelter, and eventually bodily integrity. While presenting the correct passwords had sufficient intimidation value to advance one's movement towards the "mock airbase" (i.e. the feeder and/or pleasure center activation button as provided during the given timeframe).
Furthermore - regardless whether I've been historically afforded any sort of choice in how to conceptualize my own thought processes, or indeed whether to have those in the first place - any entity which has actual power to determine my state of existence (think institutions, businesses, gangs, particularly capable individuals - all sorts of autonomous corpora) has no choice but to interpret me as either a sequence of words, a sequence of numbers, or some other symbol sequence (e.g. the ones printed on my identity documents, the ones recorded in my bank's database, or the metadata gathered from my online represence).
My first-person perspective, being constitutionally inaccessible to such entities, does not have practical significance to them, and is thus elided from the process of "self-determination". As far as anyone's concerned, "I" am a particular sequence of that anyone's preferred representational symbols. For example if you relate to me on the personal level, I will probably be a sequence of your emotions. Either way, what I may hypothetically be to myself is practically immaterial and therefore not a valid object of communication.
Excellent point. See also the failure of Sapir-Whorf to prove that language determines thought. I think we have plenty of evidence that, while language can influence thought, it is not thought itself. Many people invested in AI are happy to throw out decades of linguistic evidence that language and thought are separate.
I was with you up to the last sentence. By what reasoning do you claim that LLMs only consist of words? The input and output are words but all the stuff in the middle - where the magic happens - does not appear to be quite that simple.
Even if they are "simplistic projections", which I don't think is the correct way to think about it, there's no reason that more LLM thoughts in middle layers can't also exist and project down at the end. Though there might be efficency issues because the latent thoughts have to be recomputed a lot.
Though I do think in human brains it's also an interplay where what we write/say also loops back into the thinking as well. Which is something which is efficient for LLMs.
> But raising kids, I can clearly see that intelligence isn't just solved by LLMs
Funny, I have the opposite experience. Like early LLMs kids tend to give specific answers to the questions they don't understand or don't really know or remember the answer to. Kids also loop (give the same reply repeatedly to different prompts), enter highly emotional states where their output is garbled (everyone loves that one), etc. And it seems impossible to correct these until they just get smarter as their brain grows.
What's even more funny is that adults tend to do all these things as well, just less often.
The lack of results so far. The observable behavior exhibits many uncanny similarities but also clearly has missing pieces.
As the person you initially responded to said, observing children growing up should make it obvious.
Or if we shift to stating the obvious, there's the minor detail that the vast majority of architectures lack the ability to learn during inference. That's one of the basic things that biological systems are capable of.
LLMs and human brains are both just mechanisms. Why would one mechanism a priori be capable of "learning abstract thought", but no others?
If it turns out that LLMs don't model human brains well enough to qualify as "learning abstract thought" the way humans do, some future technology will do so. Human brains aren't magic, special or different.
Human brains aren’t magic in the literal sense but do have a lot of mechanisms we don’t understand.
They’re certainly special both within the individual but also as a species on this planet. There are many similar to human brains but none we know of with similar capabilities.
They’re also most obviously certainly different to LLMs both in how they work foundationally and in capability.
I definitely agree with the materialist view that we will ultimately be able to emulate the brain using computation but we’re nowhere near that yet nor should we undersell the complexity involved.
When someone says "AIs aren't really thinking" because AIs don't think like people do, what I hear is "Airplanes aren't really flying" because airplanes don't fly like birds do.
This really shows how imprecise a term 'thinking' is here. In this sense any predictive probabilistic blackbox model could be termed 'thinking'. Particularly when juxtaposed against something as concrete as flight that we have modelled extremely accurately.
that depends, if you explain the rules of the game you're playing and give the dice a goal to win the game, do they adjust the numbers they reveal according to the rules of the game?
That's a fallacy of denial of the antecedent. You are inferring from the fact that airplanes really fly that AIs really think, but it's not a logically valid inference.
Observing a common (potential) failure mode is not equivalent to asserting a logical inference. It is only a fallacy if you "P, therefore C" which GP is not (at least to my eye) doing.
Whenever someone paraphrases a folksy aphorism about airplanes and birds or fish and submarines I suppose I'm meant to rebut with folksy aphorisms like:
"A.I. and humans are as different as chalk and cheese."
As aphorisms are a good way to think about this topic?
I agree we shouldn't undersell or underestimate the complexity involved, but when LLM's start contributing significant ideas to scientists and mathematicians, its time to recognize that whatever tricks are used in biology (humans, octopuses, ...) may still be of interest and of value, but they no longer seem like the unique magical missing ingredients which were so long sought after.
From this point on its all about efficiencies:
modeling efficiency: how do we best fit the elephant, with bezier curves, rational polynomials, ...?
memory bandwidth training efficiency: when building coincidence statistics, say bigrams, is it really necessary to update the weights for all concepts? a co-occurence of 2 concepts should just increase the predicted probability for the just observed bigram and then decrease a global coefficient used to scale the predicted probabilities. I.e. observing a baobab tree + an elephant in the same image/sentence/... should not change the relative probabilities of observing french fries + milkshake versus bicycle + windmill. This indicates different architectures should be possible with much lower training costs, by only updating weights of the concepts observed in the last bigram.
ofc, and probably will never understand because of sheer complexity. It doesn't mean we can't replicate the output distribution through data. Probably when we do in efficient manners, the mechanisms (if they are efficient) will be learned too.
As the result, all living cells with DNA emit coherent (as in lasers) light [2]. There is a theory that this light also facilitates intercellular communication.
Chemical structures in dendrites, not even neurons, are capable to compute XOR [3] which require multilevel artificial neural network with at least 9 parameters. Some neurons in brain have hundredths of thousands of dendrites, we are now talking of millions of parameters only in single neuron's dendrites functionality.
So, while human brains aren't magic, special or different, they are just extremely complex.
Imagine building a computer with 85 billions of superconducting quantum computers, optically and electrically connected, each capable of performing computations of a non-negligibly complex artificial neural network.
All three appear to be technically correct, but are (normally) only incidental to the operation of neurons as neurons. We know this because we can test what aspects of neurons actually lead to practical real world effects. Neurophysiology is not a particularly obscure or occult field, so there are many many papers and textbooks on the topic.(And there's a large subset you can test on yourself, besides, though I wouldn't recommend patch-clamping!)
> We know this because we can test what aspects of neurons actually lead to practical real world effects.
Electric current is also quantum phenomena, but it is also very averaged in most circumstances that lead to practical real world effects.
What is wonderful here is that contemporary electronics wizardry that allowed us to have machines that mimic some of thinking, also is very concerned of the quantum-level electromagnetic effects at the transistor level.
On reread, if your actual argument is that SNN are surprisingly sophisticated and powerful, and we might be underestimating how complex the brain's circuits really are, then maybe we're in violent agreement.
They are extremely complex, but is that complexity required for building a thinking machine? We don't understand bird physiology enough to build a bird from scratch, but an airplane flies just the same.
The complexities of contemporary computers and complexities of computing-related infrastructure (consider ASML and electricity) are orders of magnitudes higher than what was needed for first computers. The difference? We have something that mimics some aspects of (human) thinking.
How complex our everything computing-related should be to mimic thinking (of humans) little more closely?
Are we not just getting lost in semantics when we say "fly"? An airplane does not at all perform the same behavior as a bird. Do we say that boats or submarines "swim"?
Planes and boats disrupt the environments they move through and air and sea freight are massive contributors to pollution.
(Motors and human brains are both just mechanisms, the reason one is a priori capable of learning abstract thought and not the other ?)
While I agree to some extent with the materialistic conception, the brain is not an isolated mechanism, but rather the element of a system which itself isn't isolated from the experience of being a body in a world interacting with different systems to form super systems.
The brain must be a very efficient mechanism, because it doesn't need to ingest the whole textual production of the human world in order to know how to write masterpieces (music, litterature, films, software, theorems etc...). Instead the brain learns to be this very efficient mechanism with (as a starting process) feeling its own body sh*t on itself during a long part of its childhood.
I can teach someone to become really good at producing fine and efficient software, but on the contrary I can only observe everyday that my LLM of choice keeps being stupid even when I explain it how it fails. ("You're perfectly right !").
It is true that there's nothing magical about the brain, but I am pretty sure it must be stronger tech than a probabilistic/statistical next word guesser (otherwise there would be much more consensus about the usability of LLMs I think).
I'm not arguing that human brains are magic. the current AI models will probably teach us more about what we didn't know about intelligence than anything else.
Thanks for the link, I haven't seen this before and it's interesting.
I don't think the version of self awareness they demonstrated is synonymous with subjective experience. But same thing can be said about any human other then me.
Damn, just let me believe all brains are magical or I'll fall into solipsism.
> LLMs and human brains are
both just mechanisms. Why would one mechanism a priori be capable of "learning abstract thought", but no others?
“Internal combustion engines and human brains are both just mechanisms. Why would one mechanism a priori be capable of "learning abstract thought", but no others?”
The question isn't about what an hypothetical mechanism can do or not, it's about whether the concrete mechanism we built does or not. And this one doesn't.
The general argument you make is correct, but you conclusion "And this one doesn't." is as yet uncertain.
I will absolutely say that all ML methods known are literally too stupid to live, as in no living thing can get away with making so many mistakes before it's learned anything, but that's the rate of change of performance with respect to examples rather than what it learns by the time training is finished.
What is "abstract thought"? Is that even the same between any two humans who use that word to describe their own inner processes? Because "imagination"/"visualise" certainly isn't.
> no living thing can get away with making so many mistakes before it's learned anything
If you consider that LLMs have already "learned" more than any one human in this world is able to learn, and still make those mistakes, that suggests there may be something wrong with this approach...
To a limited degree, they can compensate for being such slow learners (by example) due to the transistors doing this learning being faster (by the wall clock) than biological synapses to the same degree to which you walk faster than continental drift. (Not a metaphor, it really is that scale difference).
However, this doesn't work on all domains. When there's not enough training data, when self-play isn't enough… well, this is why we don't have level-5 self-driving cars, just a whole bunch of anecdotes about various different self-driving cars that work for some people and don't work for other people: it didn't generalise, the edge cases are too many and it's too slow to learn from them.
So, are LLMs bad at… I dunno, making sure that all the references they use genuinely support the conclusions they make before declaring their task is complete, I think that's still a current failure mode… specifically because they're fundamentally different to us*, or because they are really slow learners?
* They *definitely are* fundamentally different to us, but is this causally why they make this kind of error?
But humans do the same thing. How many eons did we make the mistake of attributing everything to God's will, without a scientific thought in our heads? It's really easy to be wrong, when the consequences don't lead to your death, or are actually beneficial. The thinking machines are still babies, whose ideas aren't honed by personal experience; but that will come, in one form or another.
I'm not sure. If you see what they're doing with feedback already in code generation. The LLM makes a "hallucination", generates the wrong idea, then tests its code only to find out it doesn't compile. And goes on to change its idea, and try again.
We seem to be talking past one another. All I was talking about was the facts of how these systems perform, without any reverence about it at all.
But to your point, I do see a lot of people very emotionally and psychologically committed to pointing out how deeply magical humans are, and how impossible we are to replicate in silicon. We have a religion about ourselves; we truly do have main character syndrome. It's why we mistakenly thought the earth was at the center of the universe for eons. But even with that disproved, our self-importance remains boundless.
> I do see a lot of people very emotionally and psychologically committed to pointing out how deeply magical humans are, and how impossible we are to replicate in silicon.
This a straw man, the question isn't if this is possible or not (this is an open question), it's about whether or not we are already here, and the answer is pretty straightforward: no we aren't. (And the current technology isn't going to bring us anywhere near that)
> but that's the rate of change of performance with respect to examples rather than what it learns by the time training is finished.
It's not just that. The problem of “deep learning” is that we use the word “learning” for something that really has no similarity with actual learning: it's not just that it converges way too slowly, it's also that it just seeks to minimize the predicted loss for every samples during training, but that's no how humans learn. If you feed it enough flat-earther content, as well a physics books, an LLM will happily tells you that the earth is flat, and explain you with lots of physics why it cannot be flat. It simply learned both “facts” during training and then spit it out during inference.
A human will learn one or the other first, and once the initial learning is made, it will disregards all the evidence of the contrary, until maybe at some point it doesn't and switches side entirely.
LLMs don't have an inner representation of the world and as such they don't have an opinion about the world.
The humans can't see the reality for itself, but they at least know it exists and they are constantly struggling to understand it. The LLM, by nature, is indifferent to the world.
> If you feed it enough flat-earther content, as well a physics books, an LLM will happily tells you that the earth is flat, and explain you with lots of physics why it cannot be flat.
This is a terrible example, because it's what humans do as well. See religious, or indeed military, indoctrination. All propaganda is as effective as it is, because the same message keeps getting hammered in.
And not just that, common misconceptions abound everywhere and not just conspiracy theories, religion, and politics. My dad absolutely insisted that the water draining in toilets or sinks are meaningfully influenced by the Coriolis effect, used an example of one time he went to the equator and saw a demonstration of this on both sides of the equator. University education and lifetime career in STEM, should have been able to figure out from first principles why the Coriolis effect is exactly zero on the equator itself, didn't.
> A human will learn one or the other first, and once the initial learning is made, it will disregards all the evidence of the contrary, until maybe at some point it doesn't and switches side entirely.
We don't have any way to know what a human would do if they could read the entire internet, because we don't live long enough to try.
The only bet I'd make is that we'd be more competent than any AI doing the same, because we learn faster from fewer examples, but that's about it.
> LLMs don't have an inner representation of the world and as such they don't have an opinion about the world.
There is evidence that they do have some inner representation of the world, e.g.:
> This is a terrible example, because it's what humans do as well. See religious, or indeed military, indoctrination. All propaganda is as effective as it is, because the same message keeps getting hammered in.
You completely misread my point.
The key thing with humans isn't that they cannot believe in bullshit. They can definitely do. But we don't usually believe in both the bullshit and in the fact the BS is actually BS. We have opinions on the BS. And we, as a species, routinely die or kill for these opinions, by the way. LLM don't care about anything.
> But we don't usually believe in both the bullshit and in the fact the the BS is actually BS.
I can't parse what you mean by this.
> LLM don't care about anything.
"Care" is ill-defined. LLMs are functions that have local optima (the outputs); those functions are trained to approximate other functions (e.g. RLHF) that optimise other things that can be described with functions (what humans care about). It's a game of telephone, like how Leonard Nimoy was approximating what the script writers were imagining Spock to be like when given the goal of "logical and unemotional alien" (ditto Brent Spiner, Data, "logical and unemotional android"), and yet humans are bad at writing such characters: https://tvtropes.org/pmwiki/pmwiki.php/Main/StrawVulcan
But rather more importantly in this discussion, I don't know what you care about when you're criticising AI for not caring, especially in this context. How, *mechanistically*, does "caring" matter to "learning abstract thought", and the question of how closely LLMs do or don't manage it relative to humans?
I mean, in a sense, I could see why someone might argue the exact opposite, that LLMs (as opposed to VLMs or anything embodied in a robot, or even pure-text agents trained on how tools act in response to the tokens emitted) *only* have abstract "thought", in so far as it's all book-learned knowledge.
>> But we don't usually believe in both the bullshit and in the fact the the BS is actually BS.
> I can't parse what you mean by this.
The point is that humans care about the state of a distributed shared world model and use language to perform partial updates to it according to their preferences about that state.
Humans who prefer one state (the earth is flat) do not -- as a rule -- use language to undermine it. Flat earthers don't tell you all the reasons the earth cannot be flat.
But even further than this, humans also have complex meta-preferences of the state, and their use of language reflects those too. Your example is relevant here:
> My dad absolutely insisted that the water draining in toilets or sinks are meaningfully influenced by the Coriolis effect [...]
> [...] should have been able to figure out from first principles why the Coriolis effect is exactly zero on the equator itself, didn't.
This is an exemplar of human behavior. Humans act like this. LLMs don't. If your dad did figure out from first principles and expressed it and continued insisting the position, I would suspect them of being an LLM, because that's how LLMs 'communicate'.
Now that the what is clear -- why? Humans experience social missteps like that as part of the loss surface. Being caught in a lie sucks, so people learn to not lie or be better at it. That and a million other tiny aspects of how humans use language in an overarching social context.
The loss surface that LLMs see doesn't have that feedback except in the long tail of doing Regularized General Document Corpora prediction perfectly. But it's so far away compared to just training on the social signal, where honesty is immediately available as a solution and is established very early in training instead of at the limit of low loss.
How humans learn (embedded in a social context from day one) is very effective at teaching foundational abilities fast. Natural selection cooked hard. LLM training recipes do not compare, they're just worse in so many different ways.
Thermometers and human brains are both mechanisms. Why would one be capable of measuring temperature and other capable of learning abstract thought?
> If it turns out that LLMs don't model human brains well enough to qualify as "learning abstract thought" the way humans do, some future technology will do so. Human brains aren't magic, special or different.
> Everyone is out here acting like "predicting the next thing" is somehow fundamentally irrelevant to "human thinking" and it is simply not the case.
Nobody is. What people are doing is claiming that "predicting the next thing" does not define the entirety of human thinking, and something that is ONLY predicting the next thing is not, fundamentally, thinking.
Well, yes because thinking soon requires interacting, not just ideating. It's in the dialogue between ideation and interaction that we make our discoveries.
when LLM popped out and people started to say 'this is just markov chain on steroid and not thinking' i was a bit confused because a lot of my "thinking" is statistical too.. I very often try to solve an issue by switching a known solution with a different "probable" variant of it (tweaking a parameter)
LLMs have higher dimensions (they map token to grammatical and semantical space) .. it might not be thinking but it seems on its way we're just thinking with more abstractions before producing speech ?... dunno
I claim that all of thinking can be reduced to predicting the next thing. Predicting the next thing = thinking in the same way that reading and writing strings of bytes is a universal interface, or every computation can be done by a Turing machine.
People can claim whatever they like. That doesn't mean it's a good or reasonable hypothesis (especially for one that is essentially unfalsifible like predictive coding).
The problem is that we don’t have a good understanding of what “thinking” really is, and those parts of it we think we do understand involve simple things done at scale (electrical pulses on specific pathways, etc).
It is not unreasonable to suspect differences between humans and LLMs are differences in degree, rather than category.
I'm not trying to advance a testable hypothesis. If you think the unfalsifiability of my claim is a problem, you haven't understood what I'm trying to do.
My claim is that the two concepts are indistinguishable, thus equivalent. The unfalsifiability is what makes it a natural equivalence, the same as in the other examples I gave.
IMHO, you should. The opponent does not have an alternative definition of thinking that would have a prediction power matching the token prediction. Whatever they are thinking thinking is is a strictly worse scientific theory.
A motorcycle is not "sprinting" and an LLM is not "thinking". Everyone would agree that a motorcycle is not running but the same dumb shit is posted over and over and over on here that somehow the LLM is "thinking".
But your assertion is merely semantic. It doesn't say anything substantive.
I could also say a motorcycle "moves forward" just like a person "moves forward". Whether we use the same or different words for same or different concepts doesn't say anything about the actual underlying similarity.
And please don't call stuff "dumb shit" here. Not appropriate for HN.
A forklift is "lifting" things, despite using a completely different mechanical process as a human "lifting" things. The only real similarity between these kinds of "lifting" is the end result, something is higher up than it was before.
That’s because the motorcycle thing is too simlistic of a comparison. It doesn’t come nearly close to capturing the nuance of the whole LLM “thinking” situation.
I am not having some existential crisis, but if we get to a point where X% of humans cannot outperform “AI” on any task that humans deem “useful”, for some nontrivial value of X, then many assumptions that culture has inculcated into me about humanity are no longer valid.
What is the role of humans then?
Can it be said that humans “think” if they can’t think a thought that a non thinking AI cannot also think?
If all humans were suddenly wiped off the face of the earth, AI would go silent, and the hardware it runs on would eventually shut down.
If all AI was suddenly wiped off the face of the earth, humans would rebuild it, and would carry on fine in the meantime.
One AI researcher decades ago said something to the effect of: researchers in biology look at living organisms and wonder how they live; researchers in physics look at the cosmos and wonder what all is out there; researchers in artificial intelligence look at computer systems and wonder how they can be made to wonder such things.
It may be doing the "thinking" and could reach AGI. But we don't want it. We don't want to take a fork lift to the gym. We don't want plastic aliens showing off their AGI and asking humanity to outsource human thinking and decision-making to them.
Predicting the next token is not at all the same thing as predicting the next action in a causal chain of actions. Not even close. One is model of language tokens, the other is a model of the physical world. You can come up with all sorts of predictions that can't be expressed cleanly in natural language. And plenty of things that parse cleanly from a language perspective but are unhinged in their description of empirical reality.
When you have a thought, are you "predicting the next thing"—can you confidently classify all mental activity that you experience as "predicting the next thing"?
Language and society constrains the way we use words, but when you speak, are you "predicting"? Science allows human beings to predict various outcomes with varying degrees of success, but much of our experience of the world does not entail predicting things.
How confident are you that the abstractions "search" and "thinking" as applied to the neurological biological machine called the human brain, nervous system, and sensorium and the machine called an LLM are really equatable? On what do you base your confidence in their equivalence?
Does an equivalence of observable behavior imply an ontological equivalence? How does Heisenberg's famous principle complicate this when we consider the role observer's play in founding their own observations? How much of your confidence is based on biased notions rather than direct evidence?
The critics are right to raise these arguments. Companies with a tremendous amount of power are claiming these tools do more than they are actually capable of and they actively mislead consumers in this manner.
> When you have a thought, are you "predicting the next thing"
Yes. This is the core claim of the Free Energy Principle[0], from the most-cited neuroscientist alive. Predictive processing isn't AI hype - it's the dominant theoretical framework in computational neuroscience for ~15 years now.
> much of our experience of the world does not entail predicting things
Introspection isn't evidence about computational architecture. You don't experience your V1 doing edge detection either.
> How confident are you that the abstractions "search" and "thinking"... are really equatable?
This isn't about confidence, it's about whether you're engaging with the actual literature. Active inference[1] argues cognition IS prediction and action in service of minimizing surprise. Disagree if you want, but you're disagreeing with Friston, not OpenAI marketing.
> How does Heisenberg's famous principle complicate this
It doesn't. Quantum uncertainty at subatomic scales has no demonstrated relevance to cognitive architecture. This is vibes.
> Companies... are claiming these tools do more than they are actually capable of
Possibly true! But "is cognition fundamentally predictive" is a question about brains, not LLMs. You've accidentally dismissed mainstream neuroscience while trying to critique AI hype.
Thanks for the links! I'll have to dig into this more for sure. Looking at the bulleted summary, I'm not sure your argument is sufficiently nuanced or being made in good faith.
The article argues that the brain "predicts" acts of perception in order to minimize surprise. First of all, very few people mean to talk about these unconscious operations of the brain when they claim they are "thinking". Most people have not read enough neuroscience literature to have such a definition. Instead, they tend to mean "self-conscious activity" when they say "thinking". Thinking, the way the term is used in the vernacular, usually implies some amount of self-reflexivity. This is why we have the term "intuition" as opposed to thinking after all. From a neuronal perspective, intuition is still thinking, but most people don't think (ha) of the word thinking to encompass this, and companies know that.
It is clear to me, as it is to everyone one the planet, that when OpenAI for example claims that ChatGPT "thinks" they want consumers to make the leap to cognitive equivalence at the level of self-conscious thought, abstract logical reasoning, long-term learning, and autonomy. These machines are designed such that they do not even learn and retain/embed new information past their training date. That already disqualifies them from strong equivalence to human beings, who are able to rework their own tendencies toward prediction in a meta cognitive fashion by incorporating new information.
How does the free energy principle align with system dynamics and the concept of emergence? Yes, our brain might want to optimize for lack of surprise, but that does not mean it can fully avoid emergent or chaotic behavior stemming from the incredibly complex dynamics of the linked neurons?
FEP doesn't conflict with complex dynamics, it's a mathematical framework for explaining how self-organizing behavior arises from simpler variational principles. That's what makes it a theory rather than a label.
The thing you're doing here has a name: using "emergence" as a semantic stopsign. "The system is complex, therefore emergence, therefore we can't really say" feels like it's adding something, but try removing the word and see if the sentence loses information.
"Neurons are complex and might exhibit chaotic behavior" - okay, and? What next? That's the phenomenon to be explained, not an explanation.
This was articulated pretty well 18 years ago [0].
This essay completely misunderstands how the notion of emergence gained prominence and how people tend to actually use it. It's a straw man that itself devolves into a circular argument "embrace a reductionist epistemology because you should embrace a reductionist epistemology".
It doesn't even meaningfully engage with the historical literature that established the term, etc. If you want to actually understand why the term gained prominence, check out the work of Edgar Morin.
> can you confidently classify all mental activity that you experience as "predicting the next thing"? [...] On what do you base your confidence in their equivalence?
To my understanding, bloaf's claim was only that the ability to predict seems a requirement of acting intentionally and thus that LLMs may "end up being a component in a system which actually does think" - not necessarily that all thought is prediction or that an LLM would be the entire system.
I'd personally go further and claim that correctly generating the next token is already a sufficiently general task to embed pretty much any intellectual capability. To complete `2360 + 8352 * 4 = ` for unseen problems is to be capable of arithmetic, for instance.
> When you have a thought, are you "predicting the next thing"—can you confidently classify all mental activity that you experience as "predicting the next thing"?
So notice that my original claim was "prediction is fundamental to our ability to act with intent" and now your demand is to prove that "prediction is fundamental to all mental activity."
That's a subtle but dishonest rhetorical shift to make me have to defend a much broader claim, which I have no desire to do.
> Language and society constrains the way we use words, but when you speak, are you "predicting"?
Yes, and necessarily so. One of the main objections that dualists use to argue that our mental processes must be immaterial is this [0]:
* If our mental processes are physical, then there cannot be an ultimate metaphysical truth-of-the-matter about the meaning of those processes.
* If there is no ultimate metaphysical truth-of-the-matter about what those processes mean, then everything they do and produce are similarly devoid of meaning.
* Asserting a non-dualist mind therefore implies your words are meaningless, a self-defeating assertion.
The simple answer to this dualist argument is precisely captured by this concept of prediction. There is no need to assert some kind of underlying magical meaning to be able to communicate. Instead, we need only say that in the relevant circumstances, our minds are capable of predicting what impact words will have on the receiver and choosing them accordingly. Since we humans don't have access to each other's minds, we must not learn these impacts from some kind of psychic mind-to-mind sense, but simply from observing the impacts of the words we choose on other parties; something that LLMs are currently (at least somewhat) capable of observing.
Yes, personally I'm completely fine with the fact that LLMs don't actually think. I don't care that they're not AGI, though the hysterics about "AGI is so close now" seems silly to me. Fusion reactors and self-driving cars are just around the same corner.
They prove to have some useful utility to me regardless.
I'm an LLMs are being used in workflows they don't make sense in-sayer. And while yes, I can believe that LLMs can be part of a system that actually does think, I believe that to achieve true "thinking", it would likely be a system that is more deterministic in its approach rather than probabilistic.
Especially when modeling acting with intent. The ability to measure against past results and think of new innovative approaches seems like it may come from a system that may model first and then use LLM output. Basically something that has a foundation of tools rather than an LLM using MCP. Perhaps using LLMs to generate a response that humans like to read, but not in them coming up with the answer.
Either way, yes, its possible for a thinking system to use LLMs (and potentially humans piece together sentences in a similar way), but its also possible LLMs will be cast aside and a new approach will be used to create an AGI.
So for me: even if you are an AI-yeasayer, you can still believe that they won't be a component in an AGI.
You can make a separate model for the task, which is based on well chosen features and calibrated from actual data. Then the LLM only needs to generate the arguments to this model (extract those features from messages) and call it like a MCP tool. This external tool can be a simple Sklearn model.
A good heuristic is that if an argument resorts to "actually not doing <something complex sounding>" or "just doing <something simple sounding>" etc, it is not a rigorous argument.
And the big players have built a bunch of workflows which embed many other elements besides just "predictions" into their AI product. Things like web search, to incorporating feedback from code testing, to feeding outputs back into future iterations. Who is to say that one or more of these additions has pushed the ensemble across the threshold and into "real actual thinking."
The near-religious fervor which people insist that "its just prediction" makes me want to respond with some religious allusions of my own:
> Who is this that wrappeth up sentences in unskillful words? Gird up thy loins like a man: I will ask thee, and answer thou me. Where wast thou when I laid up the foundations of the earth? tell me if thou hast understanding. Who hath laid the measures thereof, if thou knowest? or who hath stretched the line upon it?
The point is that (as far as I know) we simply don't know the necessary or sufficient conditions for "thinking" in the first place, let alone "human thinking." Eventually we will most likely arrive at a scientific consensus, but as of right now we don't have the terms nailed down well enough to claim the kind of certainty I see from AI-detractors.
I take a offence in the idea I’m “religiously downplaying LLMs”. I pay top dollar for access to the best models because I want the capabilities to be good / better. Just because I’m documenting my experience it doesn’t mean I have an Anti-ai agenda ? I pay because I find LLMs to be useful. Just not in the way suggested by the marketing teams.
I’m downplaying because I have honestly been burned by these tools when I’ve put trust in their ability to understand anything, provide a novel suggestion or even solve some basic bugs without causing other issues.?
I use all of the things you talk about extremely frequently and again, there is no “thinking” or consideration on display that suggests these things work like us, else why would we be having this conversation if they were ?
> I’m downplaying because I have honestly been burned by these tools when I’ve put trust in their ability to understand anything, provide a novel suggestion or even solve some basic bugs without causing other issues.?
I've had that experience plenty of times with actual people...
LLMs don't "think" like people do, that much is pretty obvious. But I'm not at all sure whether what they do can be called "thinking" or not.
> The harms engendered by underestimating LLM capabilities are largely that people won't use the LLMs.
Speculative fiction about superintelligences aside, an obvious harm to underestimating the LLM's capabilities is that we could effectively be enslaving moral agents if we fail to correctly classify them as such.
> The issue is that prediction is "part" of the human thought process, it's not the full story...
Do you have a proof for this?
Surely such a profound claim about human thought process must have a solid proof somewhere? Otherwise who's to say all of human thought process is not just a derivative of "predicting the next thing"?
What would change your mind? It's an exercise in feasibility.
For example, I don't believe in time travel. If someone made me time travel, and made it undeniable that I was transported back to 1508, then I would not be able to argue against it. In fact, no one in such position would.
What is that equivalent for your conviction? There must be something, otherwise, it's just an opinion that can't be changed.
You don't need to present some actual proof or something. Just lay out some ideas that demonstrate that you are being rational about this and not just sucking up to LLM marketing.
It's fascinating when you look at each technical component of cognition in human brains and contrast against LLMs. In humans, we have all sorts of parallel asynchronous processes running, with prediction of columnar activations seemingly the fundamental local function, with tens of thousands of mini columns and regions in the brain corresponding to millions of networked neurons using the "predict which column fires next" objective to increment or decrement the relative contribution of any functional unit.
In the case of LLMs you run into similarities, but they're much more monolithic networks, so the aggregate activations are going to scan across billions of neurons each pass. The sub-networks you can select each pass by looking at a threshold of activations resemble the diverse set of semantic clusters in bio brains - there's a convergent mechanism in how LLMs structure their model of the world and how brains model the world.
This shouldn't be surprising - transformer networks are designed to learn the complex representations of the underlying causes that bring about things like human generated text, audio, and video.
If you modeled a star with a large transformer model, you would end up with semantic structures and representations that correlate to complex dynamic systems within the star. If you model slug cellular growth, you'll get structure and semantics corresponding to slug DNA. Transformers aren't the end-all solution - the paradigm is missing a level of abstraction that fully generalizes across all domains, but it's a really good way to elicit complex functions from sophisticated systems, and by contrasting the way in which those models fail against the way natural systems operate, we'll find better, more general methods and architectures, until we cross the threshold of fully general algorithms.
Biological brains are a computational substrate - we exist as brains in bone vats, connected to a wonderfully complex and sophisticated sensor suite and mobility platform that feeds electrically activated sensory streams into our brains, which get processed into a synthetic construct we experience as reality.
Part of the underlying basic functioning of our brains is each individual column performing the task of predicting which of any of the columns it's connected to will fire next. The better a column is at predicting, the better the brain gets at understanding the world, and biological brains are recursively granular across arbitrary degrees of abstraction.
LLMs aren't inherently incapable of fully emulating human cognition, but the differences they exhibit are expensive. It's going to be far more efficient to modify the architecture, and this may diverge enough that whatever the solution ends up being, it won't reasonably be called an LLM. Or it might not, and there's some clever tweak to things that will push LLMs over the threshold.
most humans in any percentile act towards the thing of someone else. most of these things are a lot worse than what the human "would originally intend". this behavior stems from 100s and thousands of nudges since childhood.
the issue with AI and AI-naysayers is, by analogy, this: cars were build to drive from A to Z. people picked up tastes and some people started building really cool looking cars. the same happens on the engineering side. then portfolio communists came with their fake capitalism and now cars are build to drive over people but don't really work because people, thankfully, are overwhelming still fighting to attempt to act towards their own intents.
Exactly. Our base learning is by example, which is very much learning to predict.
Predict the right words, predict the answer, predict when the ball bounces, etc. Then reversing predictions that we have learned. I.e. choosing the action with the highest prediction of the outcome we want. Whether that is one step, or a series of predicted best steps.
Also, people confuse different levels of algorithm.
There are at least 4 levels of algorithm:
• 1 - The architecture.
This input-output calculation for pre-trained models are very well understood. We put together a model consisting of matrix/tensor operations and few other simple functions, and that is the model. Just a normal but high parameter calculation.
• 2 - The training algorithm.
These are completely understood.
There are certainly lots of questions about what is most efficient, alternatives, etc. But training algorithms harnessing gradients and similar feedback are very clearly defined.
• 3 - The type of problem a model is trained on.
Many basic problem forms are well understood. For instance, for prediction we have an ordered series of information, with later information to be predicted from earlier information. It could simply be an input and response that is learned. Or a long series of information.
• 4 - The solution learned to solve (3) the outer problem, using (2) the training algorithm on (1) the model architecture.
People keep confusing (4) with (1), (2) or (3). But it is very different.
For starters, in the general case, and for most any challenging problem, we never understand their solution. Someday it might be routine, but today we don't even know how to approach that for any significant problem.
Secondly, even with (1), (2), and (3) exactly the same, (4) is going to be wildly different based on the data characterizing the specific problem to solve. For complex problems, like language, layers and layers of sub-solutions to sub-problems have to be solved, and since models are not infinite in size, ways to repurpose sub-solutions, and weave together sub-solutions to address all the ways different sub-problems do and don't share commonalities.
Yes, prediction is the outer form of their solution. But to do that they have to learn all the relationships in the data. And there is no limit to how complex relationships in data can be. So there is no limit on the depths or complexity of the solutions found by successfully trained models.
Any argument they don't reason, based on the fact that they are being trained to predict, confuses at least (3) and (4). That is a category error.
It is true, they reason a lot more like our "fast thinking", intuitive responses, than our careful deep and reflective reasoning. And they are missing important functions, like a sense of what they know or don't. They don't continuously learn while inferencing. Or experience meta-learning, where they improve on their own reasoning abilities with reflection, like we do. And notoriously, by design, they don't "see" the letters that spell words in any normal sense. They see tokens.
Those reasoning limitations can be irritating or humorous. Like when a model seems to clearly recognize a failure you point out, but then replicates the same error over and over. No ability to learn on the spot. But they do reason.
Today, despite many successful models, nobody understands how models are able to reason like they do. There is shallow analysis. The weights are there to experiment with. But nobody can walk away from the model and training process, and build a language model directly themselves. We have no idea how to independently replicate what they have learned, despite having their solution right in front of us. Other than going through the whole process of retraining another one.
Did anyone ever seriously buy the whole "it's transporting" BS when it was wheelbarrows? What makes you believe today's trucks are meaningfully different?
I suspect that people instinctively believe they have free will, both because it feels like we do, and because society requires us to behave that way even when we don't.
The truth is that the evidence says we don't. See the Libet experiment and its many replications.
Your decisions can be predicted from brain scans up to 10 seconds before you make them, which means they are as deterministic as an LLM's. Sorry, I guess.
It makes sense if you're desperate for free will to be real, but you really have to work for it. Especially when you add in the countless other studies showing that a lot of the reasons we give for our actions, especially in quick or ambiguous choices, are confabulationalist post-hoc constructions. Our own introspection seems mostly to consist of just "making stuff up" to justify the decisions we've already made.
I mean, a reasonable person could argue their way past all the evidence without totally denying it, but "free will" just isn't the simplest explanation that fits the available data. It's possible that free will exists in the same way it's possible that Russels teapot exists.
Maybe the point of someone else you've spoken to. Personally I don't even see how free will enters into this discussion, and if you were to ask me, I don't even think we have it as it is commonly defined/understood. But any comment on free will doesn't change the fact that Libet measured metacognition latency and nothing else.
What is the import of this to you here? Whether you have free will or you feel like you do, kinda same difference for this particular point right? It doesn't make me more human actually having free will, it is sufficient to simply walk around as if I do.
But beyond that, what do you want to say here? What is lost, what is gained? Are you wanting to say this makes us more like an LLM? How so?
The experiment raised significant questions about free will and determinism. While it suggested that unconscious brain activity precedes conscious decision-making, Libet argued that this does not negate free will, as individuals can still choose to suppress actions initiated by unconscious processes."
It's been repeated a huge number of time since, and widely debated. When Libet first did the experiment it was only like 200ms before the mind become consciously aware of the decision. More recent studies have shown they can predict actions up to 7-10 seconds before the subject is aware of having made a decision.
It's pretty hard to argue that you're really "free" to make a different decision if your body knew which you would choose 7 seconds before you became aware of it.
I mean, those long term predictions were only something like 60% accurate, but still, the preponderance of evidence says that those decisions are deterministic and we keep finding new ways to predict the outcome sooner and with higher accuracy.
"I conducted an experiment where I instructed experienced drivers to follow a path in a parking lot laid out with traffic cones, and found that we were able to predict the trajectory of the car with greater than 60% accuracy. Therefore drivers do not have free will to just dodge the cones and drive arbitrarily from the start to the finish."
Clearly, that conclusion would be patently absurd to draw from that experiment. There are so many expectation and observation effects that go into the very setup from the beginning. Humans generally follow directions, particularly when a guy in a labcoat is giving them.
> At some point, when they felt the urge to do so,
they were to freely decide between one of two buttons, operated by the
left and right index fingers, and press it immediately. [0]
Wow. TWO whole choices to choose from! Human minds tend to pre-think their choice between one of two fingers to wiggle, therefore free will doesn't exist.
> It's pretty hard to argue that you're really "free" to make a different decision if your body knew which you would choose 7 seconds before you became aware of it.
To really spell it out since the analogy/satire may be lost: You're free to refrain from pressing either button during the prompt. You're free to press both buttons at the same time. You're free to mash them rapidly and randomly throughout the whole experiment. You're free to walk into the fMRI room with a bag full of steel BB's and cause days of downtime and thousands of dollars in damage. Folks generally don't do those things because of conditioning.
If that were the only evidence I might agree that alternative explanations are as likely, but I cited only one of many studies that show similar outcomes. There are loads of other studies done with entirely different methodologies that indicate most human introspection is mostly better described as post hoc confabulism. That is to say, we don't use reason to make decisions so much as we make decisions, and then justify them with reasons. Nisbet and Wilson were showing it experimentally as far back as the late 70's. [0] It's been confirmed in different forms hundreds of times since.
Certainly we can come up with some alternative theories (like "free will") to explain it all away, but the simplest (therefore most likely correct) answer is just that we're basically statistical state machines and are as deterministic as a similar computational system.
To be clear, I'm not saying that metacognition doesn't exist. Just that I've never seen any reason to believe it's very different from current thinking models that just feed an output back in as another input.
What does it mean to say that we humans act with intent? It means that we have some expectation or prediction about how our actions will effect the next thing, and choose our actions based on how much we like that effect. The ability to predict is fundamental to our ability to act intentionally.
So in my mind: even if you grant all the AI-naysayer's complaints about how LLMs aren't "actually" thinking, you can still believe that they will end up being a component in a system which actually "does" think.