> The harms engendered by underestimating LLM capabilities are largely that people won't use the LLMs.
Speculative fiction about superintelligences aside, an obvious harm to underestimating the LLM's capabilities is that we could effectively be enslaving moral agents if we fail to correctly classify them as such.
When you have a thought, are you "predicting the next thing"—can you confidently classify all mental activity that you experience as "predicting the next thing"?
Language and society constrains the way we use words, but when you speak, are you "predicting"? Science allows human beings to predict various outcomes with varying degrees of success, but much of our experience of the world does not entail predicting things.
How confident are you that the abstractions "search" and "thinking" as applied to the neurological biological machine called the human brain, nervous system, and sensorium and the machine called an LLM are really equatable? On what do you base your confidence in their equivalence?
Does an equivalence of observable behavior imply an ontological equivalence? How does Heisenberg's famous principle complicate this when we consider the role observer's play in founding their own observations? How much of your confidence is based on biased notions rather than direct evidence?
The critics are right to raise these arguments. Companies with a tremendous amount of power are claiming these tools do more than they are actually capable of and they actively mislead consumers in this manner.
> When you have a thought, are you "predicting the next thing"
Yes. This is the core claim of the Free Energy Principle[0], from the most-cited neuroscientist alive. Predictive processing isn't AI hype - it's the dominant theoretical framework in computational neuroscience for ~15 years now.
> much of our experience of the world does not entail predicting things
Introspection isn't evidence about computational architecture. You don't experience your V1 doing edge detection either.
> How confident are you that the abstractions "search" and "thinking"... are really equatable?
This isn't about confidence, it's about whether you're engaging with the actual literature. Active inference[1] argues cognition IS prediction and action in service of minimizing surprise. Disagree if you want, but you're disagreeing with Friston, not OpenAI marketing.
> How does Heisenberg's famous principle complicate this
It doesn't. Quantum uncertainty at subatomic scales has no demonstrated relevance to cognitive architecture. This is vibes.
> Companies... are claiming these tools do more than they are actually capable of
Possibly true! But "is cognition fundamentally predictive" is a question about brains, not LLMs. You've accidentally dismissed mainstream neuroscience while trying to critique AI hype.
Thanks for the links! I'll have to dig into this more for sure. Looking at the bulleted summary, I'm not sure your argument is sufficiently nuanced or being made in good faith.
The article argues that the brain "predicts" acts of perception in order to minimize surprise. First of all, very few people mean to talk about these unconscious operations of the brain when they claim they are "thinking". Most people have not read enough neuroscience literature to have such a definition. Instead, they tend to mean "self-conscious activity" when they say "thinking". Thinking, the way the term is used in the vernacular, usually implies some amount of self-reflexivity. This is why we have the term "intuition" as opposed to thinking after all. From a neuronal perspective, intuition is still thinking, but most people don't think (ha) of the word thinking to encompass this, and companies know that.
It is clear to me, as it is to everyone one the planet, that when OpenAI for example claims that ChatGPT "thinks" they want consumers to make the leap to cognitive equivalence at the level of self-conscious thought, abstract logical reasoning, long-term learning, and autonomy. These machines are designed such that they do not even learn and retain/embed new information past their training date. That already disqualifies them from strong equivalence to human beings, who are able to rework their own tendencies toward prediction in a meta cognitive fashion by incorporating new information.
How does the free energy principle align with system dynamics and the concept of emergence? Yes, our brain might want to optimize for lack of surprise, but that does not mean it can fully avoid emergent or chaotic behavior stemming from the incredibly complex dynamics of the linked neurons?
FEP doesn't conflict with complex dynamics, it's a mathematical framework for explaining how self-organizing behavior arises from simpler variational principles. That's what makes it a theory rather than a label.
The thing you're doing here has a name: using "emergence" as a semantic stopsign. "The system is complex, therefore emergence, therefore we can't really say" feels like it's adding something, but try removing the word and see if the sentence loses information.
"Neurons are complex and might exhibit chaotic behavior" - okay, and? What next? That's the phenomenon to be explained, not an explanation.
This was articulated pretty well 18 years ago [0].
This essay completely misunderstands how the notion of emergence gained prominence and how people tend to actually use it. It's a straw man that itself devolves into a circular argument "embrace a reductionist epistemology because you should embrace a reductionist epistemology".
It doesn't even meaningfully engage with the historical literature that established the term, etc. If you want to actually understand why the term gained prominence, check out the work of Edgar Morin.
> can you confidently classify all mental activity that you experience as "predicting the next thing"? [...] On what do you base your confidence in their equivalence?
To my understanding, bloaf's claim was only that the ability to predict seems a requirement of acting intentionally and thus that LLMs may "end up being a component in a system which actually does think" - not necessarily that all thought is prediction or that an LLM would be the entire system.
I'd personally go further and claim that correctly generating the next token is already a sufficiently general task to embed pretty much any intellectual capability. To complete `2360 + 8352 * 4 = ` for unseen problems is to be capable of arithmetic, for instance.
> When you have a thought, are you "predicting the next thing"—can you confidently classify all mental activity that you experience as "predicting the next thing"?
So notice that my original claim was "prediction is fundamental to our ability to act with intent" and now your demand is to prove that "prediction is fundamental to all mental activity."
That's a subtle but dishonest rhetorical shift to make me have to defend a much broader claim, which I have no desire to do.
> Language and society constrains the way we use words, but when you speak, are you "predicting"?
Yes, and necessarily so. One of the main objections that dualists use to argue that our mental processes must be immaterial is this [0]:
* If our mental processes are physical, then there cannot be an ultimate metaphysical truth-of-the-matter about the meaning of those processes.
* If there is no ultimate metaphysical truth-of-the-matter about what those processes mean, then everything they do and produce are similarly devoid of meaning.
* Asserting a non-dualist mind therefore implies your words are meaningless, a self-defeating assertion.
The simple answer to this dualist argument is precisely captured by this concept of prediction. There is no need to assert some kind of underlying magical meaning to be able to communicate. Instead, we need only say that in the relevant circumstances, our minds are capable of predicting what impact words will have on the receiver and choosing them accordingly. Since we humans don't have access to each other's minds, we must not learn these impacts from some kind of psychic mind-to-mind sense, but simply from observing the impacts of the words we choose on other parties; something that LLMs are currently (at least somewhat) capable of observing.
The defenders and the critics around LLM anthropomorphism are both wrong.
The defenders are right insofar as the (very loose) anthropomorphizing language used around LLMs is justifiable to the extent that human beings also rely on disorder and stochastic processes for creativity. The critics are right insofar as equating these machines to humans is preposterous and mostly relies on significantly diminishing our notion of what "human" means.
Both sides fail to meet the reality that LLMs are their own thing, with their own peculiar behaviors and place in the world. They are not human and they are somewhat more than previous software and the way we engage with it.
However, the defenders are less defensible insofar as their take is mostly used to dissimulate in efforts to make the tech sound more impressive than it actually is. The critics at least have the interests of consumers and their full education in mind—their position is one that properly equips consumers to use these tools with an appropriate amount of caution and scrutiny. The defenders generally want to defend an overreaching use of metaphor to help drive sales.
Luckily for us, technologies like SQL made similar promises (for more limited domains) and C suites couldn't be bothered to learn that stuff either.
Ultimately they are mostly just clueless, so we will either end up with legions of way shittier companies than we have today (because we let them get away with offloading a bunch of work to tools they rms int understand and accepting low quality output) or we will eventually realize the continued importance of human expertise.
Or even solving problems that business need to solve, generally speaking.
This complete misunderstand of what software engineering even is is the major reason so many engineers are fed up with the clueless leaders foisting AI tools upon their orgs because they apparently lack the critical reasoning skills to be able to distinguish marketing speak from reality.
Yeah, unfortunately Marx was right about people not realizing the problem, too. The proletariat drowns in false consciousness :(
In reality, the US is finally waking up to the fact that the "golden age" of capitalism in the US was built upon the lite socialism of the New Deal, and that all the bs economic opinions the average american has subscribed to over the past few decades was completely just propaganda and anyone with half a brain cell could see from miles away that since reagonomics we've had nothing but a system that leads to gross accumulation to the top and to the top alone and this is a sure fire way (variable maximization) in any complex system to produce instability and eventual collapse.
Players of magic the gathering will say a creature "has flying" by which they mean "it can only be blocked by other creatures with reach or flying".
Newcomers obviously need to learn this jargon, but once they do, communication is greatly facilitated by not having to spell out the definition.
Just like games, the definitions in mathematics are ethereal and purely formal as well, and it would be a pain to spell them out on every occasion. It stems more from efficient communication needs then from gatekeeping.
You expect the players of the game to learn the rules before they play.
I'd say the ability to take complicated definitions and to not have to through a rigorous definition every time the ideas are referenced are, in a sense a form of abstraction, and a necessary requirement to be able to do advanced Math in the first place.
Many mathematicians do in fact teach the rules of the game in numerous introductory texts. However, you don't expect to have to explain the rules every time you play the game with people who you've established know the game. Any session would take ages if so, and in many cases the game only become more fun the more fluent the players are.
I'm not fully convinced the article makes the claim that jargon, per se, is what needs to change nor that the use of jargon causes gatekeeping. I read more about being about the inherent challenges of presenting more complicated ideas, with or without jargon and the pursuit of better methods, which themselves might actually depend on more jargon in some cases (to abstract away and offload the cognitive costs of constantly spelling out definitions). Giving a good name to something is often a really powerful way to lower the cognitive costs of arguments employing the names concept. Theoretics in large part is the hunt for good names for things and the relationships between them.
You'd be hard pressed to find a single human endeavor that does not employ jargon in some fashion. Half the point of my example was to show that you cannot escape jargon and "gatekeeping" even in something as silly and fun as a card game.
The article does not complain about notation. It describes how the different fields of mathematics are so deep and so abstract that it’s hard to understand them as a professional mathematician in a different field. That’s a hard problem worthy of discussion, but as the article says, it’s not as much a problem of notation or of explanations, rather than it’s just intrinsically difficult and complex because these are abstract and deep fields.
The only thing that sentence says is that it’s impossible to understand math without understanding the language of math and how it is constructed. Not sure how that is controversial or gatekeeping. If you are annoyed at that comment saying “learn” instead of “be taught”, I think that’s a pedantic argument because the argument wasn’t about that at all.
Again, learning notation is part of the process of learning math. No one is gatekeeping anything, at no point you need to do an exam or magically be aware of notation that you never saw. Every book and every class will define new notation at the beginning, in most cases they will do so even when there’s no new notation. I am not sure what your argument is.
That’s a very good gate to keep. Some things are just meant to be gatekept so that the cranks and dilettantes that wastes everyone’s time can stay far outside.
As someone who has always struggled with mathematics at the calculational level, but who really enjoys theorems and proofs (abstract mathematics), here are some things that help me.
1. Study predicate logic, then study it again, and again, and again. The better and more ingrained predicate logic becomes in your brain the easier mathematics becomes.
2. Once you become comfortable with predicate logic, look into set theory and model theory and understand both of these well. Understand the precise definition of "theory" wrt to model theory. If you do this, you'll have learned the rules that unify nearly all of mathematics and you'll also understand how to "plug" models into theories to try and better understand them.
3. Close reading. If you've ever played magic the gathering, mathematics is the same thing--words are defined and used in the same way in which they are in games. You need to suspend all the temptation to read in meanings that aren't there. You need to read slowly. I've often only come upon a key insight about a particular object and an accurate understanding only after rereading a passage like 50 times. If the author didn't make a certain statement, they didn't make that statement, even if it seems "obvious" you need to follow the logical chain of reasoning to make sure.
4. Translate into natural english. A lot of math books will have whole sections of proofs and
/or exercises with little to no corresponding natural language "explainer" of the symbolic statements. One thing that helps me tremendously is to try and frame any proof or theorem or collection of these in terms of the linguistic names for various definitions etc. and to try and summarize a body of proofs into helpful statements. For example "groups are all about inverses and how they allow us to "reverse" compositions of (associative) operations--this is the essence of "solvability"". This summary statement about groups helps set up a framing for me whenever I go and read a proof involving groups. The framing helps tremendously because it can serve as a foil too—i.e. if some surprising theorem contravene's the summary "oh, maybe groups aren't just about inversions" that allows for an intellectual development and expansion that I find more intuitive. I sometimes think of myself as a scientist examining a world of abstract creatures (the various models (individuals) of a particular theory (species))
5. Contextualize. Nearly all of mathematics grew out of certain lines of investigation, and often out of concrete technical needs. Understanding this history is a surprisingly effective way to make many initially mysterious aspects of a theory more obvious, more concrete, and more related to other bits of knowledge about the world, which really helps bolster understanding.
I would also like to see a custom background image. And floating text or bubbles that follows your cursor. And maybe play your favorite song in the background. And a "best friends" list (oh substack already has that basically!)
What people don't like about LLM PRs is typically:
a. The person proposing the PR usually lacks adequate context and so it makes communication and feedback, which are essential, difficult if not impossible. They cannot even explain the reasoning behind the changes they are proposing,
b. The volume/scale is often unreasonable for human reviewed to contend with.
c. The PR may not be in response to an issue but just the realization of some "idea" the author or LLM had, making it even harder to contextualize.
d. The cost asymmetry, generally speaking is highly unfavorable to the maintainers.
At the moment, it's just that LLM driven PRs have these qualities so frequently that people use LLM bans as a shorthand since writing out a lengthy policy redescrbiing the basic tenets of participation in software development is tedious and shouldn't be necessary, but here we are, in 2025 when everyone has seemingly decided to abandon those principles in favor of lazyily generating endless reams of pointless code just because they can.
The harms engendered by underestimating LLM capabilities are largely that people won't use the LLMs.
The harms engendered by overestimating their capabilities can be as severe as psychological delusion, of which we have an increasing number of cases.
Given we don't actually have a good definition of "thinking" what tack do you consider more responsible?
reply