This stuff drives me up the wall for a few reasons.
The main one is that I’ve never heard anyone describe in any detail how the technology will work. I’ve worked in big shops running big online ML systems and oh boy, do they need a lot of diaper-changing to even stay running. If all the ops folks went on vacation it’s a “hours vs days” not “weeks vs months” question of how soon it would fall over. Some log rotate thing gets stuck, and away we go with cascading failures.
So how, in some detail, do we get from big-ass mixture model of transformers to even self-operation? That’s got to be a pre-requisite for self-improvement right? inb4 “The risk is so great the details don’t matter because not impossible”, hmmm, no. Extreme tail risk isn’t interesting when addressing it comes at the cost of driving immediate, overwhelmingly likely risk of nightmare outcomes through the roof. Consolidating the alignment and steering of big models into the hands of a few CEOs, governments captured by them, or both is a clear and present danger of horrifying proportions. I take mitigate that over extreme tail risk sure as Tuesday and taxes.
Then there’s the reality of AI development thus far, which is that it comes in stops and starts. Clearly the past is no guarantee of the future, but it seems a damned sight better as a prior than the log scale and a ruler methodology employed by Yud or whoever.
Smart people talk like this is serious. There are even some elite practitioners who talk kinda like this (Hinton, Karpathy on Lex, some others). And I don’t want to be walking around with my head up my ass on it, so I’ll be grateful to anyone who wants to set me straight.
But if there’s an explanation somewhere, by a credible practitioner, of how this could actually happen in some actual technical detail, I haven’t found it.
I think that needs to be in the footnotes the next time the mainstream press starts publishing blog posts off LessWrong as consensus.
> I’ve worked in big shops running big online ML systems and oh boy, do they need a lot of diaper-changing to even stay running. If all the ops folks went on vacation it’s a “hours vs days” not “weeks vs months” question of how soon it would fall over. Some log rotate thing gets stuck, and away we go with cascading failures.
Ever worked somewhere where all the good devs have left and it's outsourced to a bunch of cheap labour? They find ways to keep it running for years. Maybe not doing the way you'd like (log rotates are stuck? Better turn off the logging system), but they make it work. Ops people like to overestimate their own importance like everyone else.
> So how, in some detail, do we get from big-ass mixture model of transformers to even self-operation? That’s got to be a pre-requisite for self-improvement right?
No? Model could propose a bunch of changes to its own code and let the human developers take care of deploying it. Of course the developers could theoretically review those changes carefully... but let's be real, they won't. In the worst case a human-level-intelligent AI could certainly email the outsourced ops staff and ask them to deploy tag x. More generally, it can just do whatever a new dev would do - read your wiki and figure out which commands to run from that. If there's an error, look up old email threads on it, or similar errors online, or try to match up a traceback against source code and look for common errors. We expect fresh grads to do this stuff all the time, it's really not that hard.
> Then there’s the reality of AI development thus far, which is that it comes in stops and starts. Clearly the past is no guarantee of the future, but it seems a damned sight better as a prior than the log scale and a ruler methodology employed by Yud or whoever.
WTF kind of logic is this? Daily weather goes up and down, so clearly extrapolating any kind of long-term trend in temperature is silly.
So there’s a middling sci-fi novel from maybe a decade ago trying to tell the AI Takeoff story called Avogadro Corp. It’s not exactly Heinlein but it’s a fun little yarn.
The basic premise is that Google pardon me Avogadro Corp is doing NLP email copy-editing, but it takes too many compute resources so in desperation the TL hand wave sets it loose to secure its own budget, and being good at writing convincing emails, well, one thing leads to another.
Now this is very hand-wavy and involves a lot of magical thinking and suspension of disbelief. But that’s ok, it’s a pulpy sci-fi novel for fun.
Now this fictional scenario is dramatically more specific, more detailed, and frankly more believable than anything anyone has linked on this thread.
And I have no issue with folks wanting to LARP this sort of thing in a capital-R Rationalist forum somewhere: it’s a premise that has fascinated sci-fi fans since Asimov wrote his 3 Laws and probably before that.
But when it starts hitting the mainstream press, K Street, freaking out everyday people, and being used to turbocharge fear-driven land-grabs around intellectual property precedent?
Yeah, it’s time to do better than “But what if I Have No Mouth and I Must Scream Was Real man?!?!”
Scorn is not an argument. I really don't see which part you think would actually be hard for this hypothetical AI to do, if there even is such a part. The whole premise is that it has better-than-human intelligence, so it can do anything that e.g. a physically disabled human employee could do, and such people are already able to improve AIs. Maybe put more effort into defining and explaining where you think that barrier is rather than expounding in detail how you think anyone who disagrees with you is the kind of person you enjoy bullying.
I apologize for coming off scornful, the last thing I read before replying was “WTF kind of logic is this” and I kinda said, ok, that’s the tone on this sub-thread. I’ll watch my snark but I’ll also admonish you to only take the vibe places you want to go.
My whole premise is that it doesn’t have better than human intelligence. Everyone glosses over how we go from ChatGPT 4 failing its own unit tests to geometrically self-improving hyper-intelligence.
The burden is not on skeptics of paperclip optimizers in the near term to prove it can’t happen. That which is not prohibited by the laws of physics is admitted by the laws of physics. Of course it’s possible in the abstract.
The burden on anyone advocating positions being taken seriously by the public, by government, by industry, etc. to show that these arguments are in the public welfare, and I contend that they are not.
> My whole premise is that it doesn’t have better than human intelligence. Everyone glosses over how we go from ChatGPT 4 failing its own unit tests to geometrically self-improving hyper-intelligence.
Obviously if it can't improve its code then it can't improve its code. But you seem to think there's some barrier that means that even if it was a better-than-human programmer (implicitly for general-intelligence reasons), it wouldn't be able to geometrically self-improve. And it's not at all obvious (at least to me) what that barrier is. We can already throw ChatGPT or equivalent at a codebase, ask it to suggest improvements to the code, and it does; the hit rate isn't great, but it's already good enough that people are incorporating it into their workflows.
As far as I can see, that's enough. Yes, people who are worried about this don't spend a lot of time going through the details, just as people who are worried about the military staging a coup don't spend a lot of time going into how they will get weapons or coordinate their movements or what have you. Those things are pretty complex, but they're ultimately routine; we are right to gloss over them, and if someone thinks that those things make a coup impossible, the burden would be on them to explain why (maybe there's some specific protocol that means they can't get access to weapons without authorisation from the legitimate authorities - but you'd need to explain what that protocol actually is and justify why you expect it to be wartertight).
Your argument seemed to be that babysitting the kind of complex code that we use for AI is not merely complex, but so complex that it requires some unique human ability that we would not expect a ChatGPT-like AI to possess (implicitly, that it requires something a lot more difficult than making improvements to code). And I just don't see that. Put it this way: we would never expect a human worker at these companies who was smart enough to make improvements to the core AI code to get stuck when it comes to actually running it. So why would it be an issue for the AI? I do think the burden is on you here given that the whole premise is that the AI is smart enough to improve its own code (you're welcome to argue that it's impossible for an AI to ever get smart enough to improve its own code - but in that case all this talk about operational complexity is beside the point).
to Cyberdyne Systems Model T-100. That’s a bit of a cheeky oversimplification of a modern decoder, but the LLaMA2 code is there for anyone to read, it’s kinda that. It’s not GPT-4, there’s clearly some scale or some special sauce or both on the way from here to there, but it’s damned close.
LLMs aren’t even a particularly great example of “adversarial” AI: AlphaZero is a way better example of AI kicking the shit out of humans in a zero-sum scenario. If I wanted to scare people I’d be talking about the DeepMind stuff, not the OpenAI stuff.
But the real crux of my argument is that bad-faith arguments have a tendency to escape the control of those making them, and if a short-term advantage in the copyright treatment of model weights resulting from the commons is achieved by Yud posting about GPT and nuclear weapons in Time Magazine and scaring the living hell out of the lay public about a capricious digital deity in the process, that’s unlikely to be a genie that goes back in the bottle.
So I’d really like it if someone injected some: here’s how this could actually happen into the dialog.
Again you seem to have no argument beyond scorn. Again: what is it that you think would be concretely hard for an AI in that position? In most tech companies a mid-level manager or senior dev can send an email and "make things happen" and I suspect most of them wouldn't know or care how exactly those things got done. But sure, no-one can explain exactly how an AI could send an email, anyone who thinks one could must be a nerd who's reading too much pulp sci-fi amirite.
Just a bystander but you just quoted this:
> My whole premise is that it doesn’t have better than human intelligence. Everyone glosses over how we go from ChatGPT 4 failing its own unit tests to geometrically self-improving hyper-intelligence.
Then did exactly what the quote is talking about, assuming we achieve better than human intelligence.
> But you seem to think there's some barrier that means that even if it was a better-than-human programmer (implicitly for general-intelligence reasons), it wouldn't be able to geometrically self-improve.
The question is how exactly do we go from a below human intelligence to an above human intelligence. Without a clear path to that reality, making trade offs in order to protect against low probability worse case scenarios which require it doesn’t look like a good deal.
The reality that right now human level intelligences have a hard time even keeping these below human intelligence systems operating seems like a useful checkpoint. Maybe we can hold off on doing distasteful things like consolidating control until they can at least wipe their own bottoms as it were.
> Then did exactly what the quote is talking about, assuming we achieve better than human intelligence.
Well, sure, because a) the article covers that b) their whole argument makes no sense if their position is that AI simply can't ever achieve better than human intelligence. If the AI is never intelligent enough to improve its own code then none of the operational complexity stuff matters!
> The question is how exactly do we go from a below human intelligence to an above human intelligence.
The same way we got to the current level of artificial intelligence; old-fashioned hard work by smart people. The point is that, if you accept that slightly-better-than-human AI would be geometrically self-improving, then by the time we have slightly-better-than-human AI it's too late to do anything.
For years, I never understood cryptocurrencies. I didn’t understand why mainstream press promoted startups managing billions run by “visionary” people (barely?) old enough to rent a car. I understood the technical aspects, just not how it was being promoted. I see now there was little to understand… It is simply an end-run around financial regulation, where the biggest visionaries are frauds, exchanges are ridiculously insecure, transactions are costly yet slow, and the primary use is illicit activities…
It’s the same thing lately with AI… Sure, it has slightly more merit than cryptocurrency, but it doesn’t do what people think it actually does — certainly not as well either.
AI is not sentient yet people write full paragraphs to it as if it were? They even expect such in response. This is LUNACY! This is a mass delusion! Real life is not a Star Trek TNG episode with a talking computer!
It’s not an accident either. Nobody wants to look like a fool when there is a lot of money at stake. So the fly by night snake oil salespeople thrive… Some technology will be developed by overly sincere people along the way, but that was never the real goal.
I think the most interesting part of AI doomsaying is that almost all of these risks apply to biological 'systems' too. There's no reason why a human can't be born that somehow has superhuman intelligence and does all of the things these evil AI would do. Moreover, a biological superintelligence has self-replication built into the system already!
Depending on how you define biological superintelligence, these persons already exist, there's at least two people in the world (on an SD16 scale) with a 200 IQ at our current global population, my IQ is in the "above-genius" territory and there are at least ~335 people as intelligent as I am just in the United States (and that's only statistical probability, due to global labor market conditions the number is probably higher).
Do you know what really smart people do? They don't do any of the things that AI Risk says will happen, they get jobs at Google (or insert other pointless white collar job on wall street) moving widgets by 1 pixel to drive an ad engagement metric by half of a percentage and get paid $500k+ a year in total compensation. Why? Because it is a relatively low effort way to fund a lifestyle that allows enjoyment of your intellect through experiencing the world in other dimensions besides the mundane of work.
The non-biological aspect is the only reason why AI Risk has any credence at all /in our current state/, because AI isn't distracted by a desire to travel the world and explore nature, or to read all the great works of literature and write their own analysis of it, etc. Biological beings, at least at this point in time and in the probable sense, have emotions and therefore have desires that range beyond whatever machinations you expect from AI Risk.
You're right that most smart people behave this way, but there's no guarantee that all of them will. The existential risk is still there (psychopathy is a real thing too).
But something else to consider is that so far biological super intelligence doesn't seem to get all that much smarter than what humans are at generally. If somebody is guarded against you, then it's very difficult (impossible) to fool them even if you are way smarter than they are. This could indicate that maybe there's a cap to how super intelligent AI can get as well.
I have noticed that the vast majority of existential risk folks I have seen have very little experience running and maintaining large systems in production. They tend to all be on the research side of things. I think that it can be hard to understand the amount of real world friction that exists until you experience it for yourself.
I agree that a lot of the AI risk pundits are either people with no real AI experience (or very outdated, resting-on-their-laurels experience) or if they do know AI, they probably know little about the complexities of running actual large scale, business or mission critical systems.
However, in their defense, the real question is whether those types of problems are something this new class of AI could potentially excel at. I've been an AI skeptic for decades, having had real world industry experience trying to get it to solve problems, and this is the first technological advance that has actually surprised me with its capabilities.
AI Risk still doesn't rank above many current, pressing global problems for me personally, but it also doesn't seem ridiculous anymore.
Imagine the real world friction overcome to evolve a human from a single celled organism.
Now consider that a computer program can replicate and change itself orders of magnitude faster than a human generation.
I've dealt with spaghetti systems that need such maintenance and I'm unconvinced that poor devops is a major barrier to the hypothetical of runaway AI. Devops people will probably be using AI to fix these issues soon enough if they aren't already.
Conversely, being too close to the problem can cause practitioners to miss the forest for the trees.
I think in this case it's a false premise though. There are plenty of real practitioners like Joshua Bengio and Geoffrey Hinton who have sounded the alarm on these topics, and no one can call them inexperienced with real systems.
The question isn't _neural_ networks, it's _physical_ networks (and power, and cooling, and racking, and deployment, and and and). Actually marshaling enough compute to do anything is a serious real-world-moving-atoms endeavor.
We are talking about superhuman AGI. It would be able to do everything human is able to do. If human SRE is able to figure it out, AGI would too. That may include buying compute from cloud platforms as well as building it's own datacenters.
It's a self-sealing argument. Anything a human can do, it can do too, because it's superhuman. Anything a human can't do, it can do anyway, because it's superhuman. Any reason it can't is false prima facie, because it's superhuman. You can claim what you like, and who's to argue? It's superhuman!
SREs being able to maintain these systems is existence proof that neural networks are able to maintain them — neural networks in the brains of human SREs. Or do you think humans evolved innate SRE skills?
The idea that people like Geof Hinton are wrong because they don't understand SRE is pretty arrogant. SRE skills are relatively straightforward vs AI theory.
You just don't understand their arguments. Put some efforts into understanding these arguments first. You can start by reading Nick Bostrom's Superintelligence book. People like you are no different that climate change deniers pooh-poohing on scientists without understanding physics involved. Be humble. People raising AGI X-risk arguments are smarter than you and gave it a lot of thought. Put an honest efforts in understanding their arguments.
Bostrom is convincing if you treat his postulates as axiomatic, the same way he does. I don't. Do you think rereading an unfalsifiable argument will make it less unfalsifiable? Or that a naked appeal to authority counts for anything at all, except in the realm of religious faith?
One of the things that keeps me rooted is that I've worked as then employed teams of experienced DBAs to maintain "self healing databases" for last 20 years. Large but non crazy / non special databases with robust reliable technology that just need tender loving care all the time. And I'm increasingly aware as I age that I am not special and everywhere else is not that much better. So I'm with you on the dangers of self running / self improving stuff running away, vs 10k immediate risks and issues (as in, happening, not hypotheticals).
Hook up the model's output to a shell. Remember it doesn't have to be engineered to modern SRE standards, it just needs to not crash. Dirty hacks are on the table. The comparison to diaper-changing is fairly illuminating, actually: even human children, let alone adults, figure out how to take care of that kind of thing, and it fades into the background of their activities. I agree that plausible mechanism is lacking in these AI risk conversations, but ops is not the hard part IMO.
meh. "intelligence" and "artificial intelligence" are marketing terms. According to Nils Nilsson, McCarthy and Minsky picked the term so the Dartmouth Summer Research Project on Artificial Intelligence would not be constrained by associations with "Cybernetics." McCarthy thought it (Cybernetics) too constrained because a: it traditionally focused on analog solutions and b: he thought Norbert Wiener was a bit pushy.
I asked Minsky about this a couple decades ago and it was the only time he didn't tell me I was asking a stupid question. Apparently, after the conference, the name stuck because they found it easier to get research grants with the more "neutral" sounding name.
I wonder how many people would trust "AI" if it was called "distributed indexing" or "stochastic generation."
All that matters is if they are effective terms or not. That's most of the point of the article; how to constructively inquire about whether giant LLMs might be effective at solving novel problems, perhaps better than humans can, in ways that matter to humans.
Since we don't have a formal model of how systems like humans work we can't just plug numbers into equations or generate proofs that LLMs and other ML systems will or will not be able to control other systems better than humans, so we revert to less-formal concepts as best as we can. The formal AI safety folks (MIRI et al) already tried the formal approach; by their own admission it's hopeless and they poured a lot of time and bright minds into it and we won't be able to formally answer questions about intelligence via decision theory or alignment before we have actually-intelligent systems interacting with us. What will happen at that point? We don't know and can only use imprecise concepts to make educated guesses. If we don't make honest attempts at this we run the risk of being blindsided by unexpectedly capable AI/ML systems.
This is why a large number of AI alignment researchers recommend pausing AI/ML systems at their current ability before they have a chance at being superhuman on most tasks. I wouldn't trust "distributed indexing" or "stochastic generation" if it could beat me at any particular skill of my choosing like similar systems can beat any of us at Chess or Go or Poker.
Intelligence has a pretty clear meaning. Of course you will have a hard time nailing down a definition but that's also true for simple words like "chair" or "car".
Artificial intelligence is also a pretty good name for a pretty clear concept: intelligence that has been artificially created. Any new ambiguity here comes from the ambiguity inherent in the word "artificial". Artifivial intelligence has a large range from if/else, linear regression and decision trees over ML methods to god-like superintelligence. That is not a problem, I think it's good that we have a term for this. I don't like when people try to reduce artificial intelligence to a smaller subset of these things. We should certainly come up with more precise terms in addition, but I think artificial intelligence is a very useful terminology and concept, not a marketing term.
> Any new ambiguity here comes from the ambiguity inherent in the word "artificial".
Not sure I agree here. Seems to me the ambiguity comes from the word "intelligence". As in, does that if/else clause indicate "intelligence" that's been programmed in. Of course we've created more terms like Artificial General Intelligence[1] to try to clear up the muddy definition of what constitutes intelligence.
I think our fundamental challenge as society begins to try to understand what AI is comes down to what "intelligence" is. As you said, we all know what it is, but can't define it easily. If I write some really clever code, is the system running it intelligent? Is some of my intelligence imbued into that system, or does it have some of its own? At what point does a stochastic process approach intelligence - when it has enough data? Does that mean babies aren't born with intelligence, but acquire it as they learn more? So many questions, good questions, necessary questions.
> does that if/else clause indicate "intelligence" that's been programmed in
Yes, systems built with if/else rules are intelligent, in that they reproduce some heuristic rule that someone had to think about. I think the term "expert system" is kind of appropriate for them
Of course they are not too intelligent, but, they are more intelligent than a rock, say. (unless this rock happen to be a chip running some neural network)
Agree fully - I've always used the term "expert system" as well, for the same distinction. An expert system operates on rules, created by a more "creatively intelligent" entity.
Though when we talk about AI now, we are talking about what the _public_ thinks "intelligence" means. This really muddies the conversation, because these technical distinctions we are making now fall completely flat. I think when you tell random people that we have created "artificial intelligence", they think we mean what I am referring to above as "creative intelligence", not "it executes an if/else rule" intelligent. So as AI becomes a topic in the mainstream, our conversations are likely to be completely misunderstood because of the colloquial meaning of the technical terms we use.
I have been working in human and artificial intelligence for several years now. It is not at all apparent to me that intelligence has a clear meaning.
I think the definitional problem is significant, but the problem is even more fundamental than that. A couple of years ago, I witnessed a heated disagreement at a conference about whether intelligence required learning or not.
Does intelligence require embodiment? Perception? Sensation? Agency? Intentionality? Cognition? Metacognition? Other higher cognitive/executive functions?
These concepts are not just to embellish the definition a bit. They are fundamental to what intelligence is.
A plausible definition you could find in a journal article might be: An embodied agent that can intentionally solve problems using information obtained from the external environment plus introspection, by way of various executive functions, and then affect the external environment accordingly.
Some philosophers and researchers would agree, and some would disagree, because of what they each think intelligence is.
> but I think artificial intelligence is a very useful terminology and concept, not a marketing term.
I agree that the term is fine. But it is absolutely also used as a nonsense marketing term. ChatGPT or one of the other OpenAI models was calling itself an "advanced AI system" or some such bullshit. The term itself is not to blame, but people definitely (mis)use it that way.
Why does there have to be one specific combination of these aspects that yields the one, true "intelligence"? The truth is that it is simply an extremely extremely broad spectrum. Any agent that performs useful computations can be argued to be intelligent in some way. It does not make sense to argue endlessly in an attempt to lift this lower bound. Let's just acknowledge that slime molds, bacteria and NPCs have some form of intelligence so we can move on and define more specific and useful types of intelligences: Embodied intelligence, social intelligence, learning intelligence, general intelligence etc. Of course these will always be vague spectra, as nailing anything clearly down is extremely technical.
Organizing things in sets is nice and all, but it is usually ill-defined on the boundaries and when it leads to endless discussions it's better to just think in terms of functionals. Some abstract functional tells you how intelligent, how embodied or how learning a certain agent is, according to some scale.
I'd strongly disagree that intelligence has a clear meaning. Depending on who you ask, you can get answers anywhere from "slime molds are intelligent" (they can hunt for food, solve mazes, etc) to "only humans are intelligent".
The problem with human intelligence is saying 'if a human can do it, it is intelligence' doesn't break down the problem space correctly. It also leaves potential gaps where a system could have a form of intelligence humans do not leading to humans misjudging that systems capabilities abd that could lead to disaster (common AI risk scenario).
Most people you ask aren't going to have an opinion that is worth voicing even if they do happen to voice it. People love to think in binary, yes/no situations, but reality isn't kind enough to present many of those in complex situations. Instead most systems live in a gradient. Sorties paradox becomes our primary means of argument and not a varying set of classification systems that break down the different pieces that belong to systems that have intelligence.
Myself, I consider both examples intelligent, but one has vastly more capabilities.
Come on. We did this already, almost certainly long before you were born. Even the instrumentalists don't like it!
This should begin with definitions of the meaning of the terms ‘machine’ and ‘think’. The definitions might be framed so as to reflect so far as possible the normal use of the words, but this attitude is dangerous. If the meaning of the words ‘machine’ and ‘think’ are to be found by examining how they are commonly used it is difficult to escape the conclusion that the meaning and the answer to the question, ‘Can machines think?’ is to be sought in a statistical survey such as a Gallup poll.
By that reasoning "Chemistry" is just a marketing term. It was used mostly interchangeably with "Alchemy" until Georg Stahl and others began writing (without justification!) that "Chemistry" describes more a more rational science and "Alchemists" were more interested in wealth. These untrue claims still became a self-fulfilling prophecy, where theories, teachings, and practice of chemistry evolved out of the former ways of thinking to become what we know it as today.
AI, as used today, is certainly a coherent concept, and this article has justified and described it more thoroughly than most concepts I use daily. Who cares about the shaky history if it means something real today?
I think the article explains very well why "intelligence" is a useful and important concept. Namely because it is concise way to discuss abilities which are highly correlated without naming each ability individually.
Somehow the rationalist community started at empty platitudes like "correlation is not causation" yet has ended up at substantive yet amazingly wrong conclusions like "correlation is identity", amazing turn, I gotta say I did not see it coming.
This is a good discussion of concepts. However, a lot of the folks worried about superintelligence miss an important division within the concept of "intelligence." Specifically with respect to the current generation of LLMs.
Namely, there's a huge difference between the concept of intelligence and an intelligent agent.
You could make the argument that current LLMs have some level of intelligence. But (without a lot of supporting infrastructure) they are not agents, they have no built in mechanism for having any kind of intent or even short-term memory. Because they fundamentally just generate a single token at once, they don't even exist "within time" the way we normally would think of an intelligent entity as doing... they're more like a mathematical function that you can invoke on demand. Perhaps to get an "intelligent" result, but conflating that kind of intelligence with an intelligent entity or actor seems like a mistake.
To use the analogy in the OP, it would be like calling a single hydraulic piston "strong". Sure, but unless you attach it to something it's not the same kind of strong as a bodybuilder is.
OK, but the hard part was the "intelligent" part, right? If you have the "intelligent" thing, and you want to get an "intelligent agent", all you have to do at most is wrap it in a shell script that runs the equivalent of an OODA loop, e.g. AutoGPT. So we should expect a "superintelligent agent" to appear pretty much the next day after a "superintelligence" is created.
OK, but the hard part was the "intelligent" part, right? If you have the "intelligent" thing, and you want to get an "intelligent agent", all you have to do at most is wrap it in a shell script that runs the equivalent of an OODA loop, e.g. AutoGPT.
No, not really.
Neither AutoGPT or any similar actually work. And the only justification for expecting them to work the future is same naively extrapolation more of one kind of quality will automatically give a thing more of a correlated quality.
The humming bird building airplanes: "If we continue to increase the capacity of these flying machines, eventually they will have 'super-bird abilities' and at that point they will naturally steal all of nectar..."
AutoGPT doesn't work because GPT-4 is too stupid to make good plans, not because it's missing the secret sauce of "agentiness." If some future software was really great at observing, planning, and reasoning, then an AutoGPT-like wrapper would make it good at autonomously doing things, too.
Perhaps, I don't discount the possibility that an LLM could be used as a primitive as part of a larger more complex system that could rightfully be called intelligent.
But a script running a LLM in an OODA loop is a lot easier to monitor and control than some inscrutable "ghost in the parameters" that a lot of AI alarmists seem worried about. You can actually see a transcript of its "thoughts", for one!
We already have self-replicating scripts that can prove very hard to detect and destroy, even without the benefit of LLM-powered fuzzing and social engineering.
As someone who has worked/is working on building LLM driven agents, this allegory basically reads to me like:
1. I ask a LLM for paperclips.
2. ~Magic happens~.
3. Humanity is doomed.
Even if a LLM was good enough to drive a system like this (current ones are not, by a long ways), there's a million reasons it would not and could not play out this way.
I think (from the outside looking in) many lesswrongers are slowly coming around from the "paperclipper" scenario to something more gradual, in response to how the tech seems to be developing currently. More along the lines of "we hand over bigger and bigger parts of running the world to AI systems and this gradually leads to less and less human involvement (until such time as the intelligent systems that run the world see humans as an unnecessary interference that can be done away with)". Something like that at least.
I mean, to me this sounds like "some christians are coming around to the fact that cosmology is mostly correct about the shape of the universe, but they want us to say that God created the big bang since you can't explain it yet."
Which is to say, right or wrong, it's a narrative in search of confirmation more than it's an actually rational understanding of what is real.
IMHO you're doing it a disservice by calling it a "narrative" and comparing to christian cosmology. On the face of it I think ideas like "past some threshold AI can become dangerous to humans" or "AI can become more intelligent through a positive feedback loop", even if speculative, have some plausibility and are worth investigating.
You are basically chiding people for changing their mind in the face of new information, while still staying committed to some core ideas based on argumentation which is not challenged by this new information. Sure, this roughly looks like what happened to religions as science developed, but it looks just as much like how any scientific or philosophical field of inquiry makes progress.
The basic idea of the essay, I think, is that "an agent" is a thing in idea-space that a non-agent intelligence (oracle) has access to and can describe to a user.
"describe" in this context means "write code to represent"
Then the user is turning what was an oracular AI into an agentic AI. The idea here, is that if you think that you're safe because you haven't made an agent, just an oracle, that's a false sense of security. Oracles are just one user choice away from becoming an agent.
If one thinks agents are safe for some reason, that's a whole separate discussion I think.
Whenever this topic comes up, there is a lot of hand-wringing as if the nightmare scenario were still in some misty future.
It's not. AIs already control the world. They have proven indifferent to the welfare of humans. Perhaps the most important developments in recent decades were their coming to own the majority of public media, and the high court of the one of biggest jurisdictions permitting them to bribe politicians secretly: in practice, to own pet legislators.
The AIs in question are called corporations. Their working parts mostly still include (readily replaced) humans, but decreasingly so. They have learned to manufacture voting blocs to drive political movements in nominally democratic countries to support their goals. By analogy to the paperclip apocalypse, they have a built-in motivation to turn all matter and energy into profit. Where their chosen method results in some service or product useful to humans, humans may benefit, but such choices may change freely, or may fail to change when the method turns out to cause net harm. Attempts at legislation to limit their power fail as they have taken control of the legislative process.
Is it too late to do anything? Probably. I for one ...
It seems to me that the bitter lesson is a counter-argument of sorts to intelligence explosion: the idea that stacking more layers is more effective than trying to be really clever about it may well persist through all levels of intelligence. To put it in another way, if the bottleneck in the progress of intelligence, or of technology in general, is not intelligence, then greater intelligence will be of limited utility in accelerating it.
The bitter lesson just says that general purpose machine learning algorithms with large amount of compute work better than domain-specific algorithms. It doesn't say that general purpose algorithms cannot be optimized. General purpose machine learning system that's able to optimize itself will lead to intelligence explosion.
I do think that there is also a "bitter lesson" in optimization: simple optimizations are usually more effective, robust and powerful than complicated ones. Also, optimization becomes progressively harder as the system trends to optimal performance, which is likely to cancel out the advantage of higher intelligence in finding them.
Also, even to the extent model architecture can be optimized, in the short term, thanks to the bitter lesson, the bottleneck seems to be less “human intelligence coming up with new ideas for optimizations” and more “compute time to test ideas on sufficiently large models”. An AI that followed the same basic thought process as humans, even if its ideas were a little better than humans’ on average, probably wouldn’t help that much.
On the other hand, an AI that in some sense ‘understands’ its own structure, which is to us almost entirely a black box, might be able to do much better. Consider OpenAI’s attempt to make GPT-4 explain the purpose of neurons in GPT-2; it basically didn’t work at all, but it would have opened a huge new frontier of research if it had.
But that understanding ability is hard to train, given the lack of any meaningful training data. If it does show up, it could act as an accelerant. But it could just as easily not show up, at least not until AI is already comfortably superhuman in the “thinking like a human” department – in which case it won’t help us reach that point in the first place.
> On the other hand, an AI that in some sense ‘understands’ its own structure, which is to us almost entirely a black box, might be able to do much better.
True... but is this a plausible capability? I don't mean in practice. I mean theoretically.
I feel we often forget about the free lunch theorem when it comes to intelligence. To us it is this magical, omnipotent thing, such that a sufficiently advanced intelligence can understand anything. But mathematically, there is no such thing as an algorithm that is better than another at everything: whatever intelligence is, it has to be intrinsically bad at certain things.
Which brings me to this thought: I think it is highly plausible that understanding one's own internal structure intimately enough to reorganize and optimize it is in fact one of these things Intelligence (with a big I: all levels of intelligence) is intrinsically bad at, both directly and indirectly. What makes me think it is plausible is that intelligence is mainly about manipulating abstractions, but reality is far too messy to be abstracted... unless you simplify it. And so it seems to me that what makes humanity incredibly powerful is the one-two punch of morseling reality into simple pieces that match our abstractions, and building with these pieces. In other words, intelligence is only effective over an ablated version of reality, and we happen to be very good at ablating... reality itself. But if the thing you are trying to understand is truly complex, you can't really ablate it to a point where it is simple enough to be understood, and basically every single level of intelligence will fail at understanding it.
There has to be some kind of tradeoff to intelligence's uncanny ability to manipulate and understand systems, and I think this is where it is. I very well could be wrong, of course.
AI risk argument is that human brain "hardware" is suboptimal as it relies on relatively slow electrochemical communication between neurons. Even with same algorithms, performance of digital intelligence will be orders of magnitude higher simply because of speed and memory capacity. Evolution rarely achieves optimal performance (optimal within constrains of physics). Evolution is iterative and tends to find local maxima. Think how we can build construction equipment thousands times stronger than any animal's muscle. And for AGI to improve itself — it's not necessary to be able to model itself with 100% accuracy. It only needs to understand itself at a high level, kinda like the way human understands its own body. Software engineers work on systems they don't completely understand all the time. E.g. AGI may notice inefficient memory use in its own implementation. It will try to fix it and run copy of itself. If it finds it indeed improves performance it will "upgrade" itself to new version, maybe just keeping one instance of previous version as a backup just in case. Now new version is smarter and is able to come up with even better optimizations. Eventually it will exhaust all possible software optimizations and will need to build new hardware to keep improving. How it will do it? Use your creativity. There are many ways. With ability to build custom hardware the only limit is now physics. It will keep improving and getting smarter until it reaches physical limits of computation. At this point it's billions time smarter than human.
> Think how we can build construction equipment thousands times stronger than any animal's muscle.
And yet the mantis shrimp can throw a punch faster than a bullet. Nature does not optimize for the same things we do. Animals did not evolve muscles as strong as some of our construction equipment, but it is hard to know whether this is because evolution is limited, or because that capability was not helpful enough to compensate for the extra fuel requirements.
> Even with same algorithms, performance of digital intelligence will be orders of magnitude higher simply because of speed and memory capacity.
Maybe. But I think you (and others) have misplaced confidence in the superiority of our technology.
Power consumption of a circuit is proportional to frequency. We have computing units that can work at GHz rates, sure, but you can't exactly cram 80 billions of these into a cubic foot unless you're planning to melt them into plasma. So we don't. Current approaches must multiplex millions of "neurons" through the same computing units, leading to effective firing rates that are not wildly different from biological neurons. If we were to switch to a different approach where each "neuron" is independent, I expect the communication speed to plummet simply because of practical considerations about energy consumption and heat dissipation.
Beyond that, I would argue that our paradigm of hardware design rests on many intentional decisions that are structurally suboptimal for intelligence, and that this is a graver problem than the brain's own inefficiencies. Take the ability that you mention to read one's own implementation and back up one's brain. This is intentional in our design. But this is not free! Unfettered read access to one's brain is a heavy internal communication requirement when you compare it to simple local communication between units, and it is naive to think it would not have a negative impact on performance. AI that can copy itself has a lower performance ceiling than a system that does not have this requirement. That's physics.
In any case, the point I'm trying to make is not about humans vs AI, it is about intelligence in general. I suspect that intelligence is excellent at simple optimization, but that it is not very good at complex optimization, and thus for any intelligence there may be a point where it starts to underperform relative to e.g. evolution.
>>It’s related to Jelinek’s Law, named after language-processing AI pioneer Frederick Jelinek: “Every time I fire a linguist, the performance of the speech recognizer goes up.” The joke is that having linguists on your team means you’re still trying to hand-code in deep principles of linguistics, instead of just figuring out how to get as big a blob of intelligence as possible and throw language at it. The limit of Jelinek’s Law is OpenAI, who AFAIK didn’t use insights from linguistics at all, and so made an AI that uses language near-perfectly.
It seems a small conjecture to note that the problem here is that "deep principles of linguistics" the linguists are attempting to hard-code, are in fact more like "intermediate principles of linguistics".
The linguists haven't approached anywhere near the deeper layers of how the brain works on language (nevermind how the Wernicke's area coordinates w/the visual cortex to generate "I see a bird over there"). So, when they code in their principles, they only screw up the deeper relationships. Of course the LLM has zero actual understanding of any abstract meaning such as "I", "see", "bird", or "over there", but it's still better off without those hardcodings to generate the phrase in a valid or hallucinated context.
If you say a whale is stronger than submarine or vice versa, it will be misleading unless you unpack the sub-claims, whereas you can say a blue whale is stronger than a killer whale or that submarine X is stronger than the Titan without unpacking it. Saying ChatGPT is more intelligent than Claude makes some sense, but there's no real point in comparing the intelligence of a person vs. an LLM any more than you would ask whether a humming bird flies better or worse than a Cessna.
> no real point in comparing the intelligence of a person vs. an LLM
Say you're deciding whether to employ an AI on a fairly open-ended family of tasks, or put out a want-ad to hire someone. What do you call what you're doing?
You’re not comparing their intelligence in some general sense; you’re comparing their ability to perform a collection of tasks that define a role. And more likely, you’ll do both because the answer is that each is better at some tasks that comprise the role. Eg, instead of hiring two positions, you’ll hire one and task the LLM with some of the work.
I said "fairly open-ended" and didn't limit the question to a mid-2023 LLM.
The point I'm aiming at is that an increasing ability to solve an open-ended range of problems affects the choices we have to make, no matter whether you object to calling it "intelligence".
Yeah, I agree, and that's one problem with focusing on what "intelligence" means: it kind of connotes, as you say, abstract or academic problems over practical shrewdness. (It doesn't have to have this restriction but people sure seem to take it that way a lot.)
earthboundkid is arguing that there's no point in making the claim that one flies better than the other because that claim breaks into many subclaims; some support one position and some support the other.
You named some subclaims that support the position that a hummingbird flies better than a Cessna.
But there are also many that support the other position: range, speed, load...
I love this dichotomy, formalists vs. nominalists. As the author argues, it's not very useful with respect to defining intelligence and LLMs.
It is however a wonderful world building detail in Anathem by Neal Stephenson. Their world history revolves around two ideologies: the syntactics and the semantics. Nominalists and Formalists. I find his predictions about a society built around this duality entertaining.
Sometimes I wish that when folks opined on such matters they would consider that there is a wealth of philosophical literature covering the same ground that generally has already anticipated your points, contradicted them, strengthened them, provided sharper terminology for discussing them, etc.
In this case, I'm thinking of natural kind terms (https://plato.stanford.edu/entries/natural-kinds/) and, less well known, Boyd's homeostatic property cluster theory. It is kind of amazing to me that this article has 310 comments and not a single person has raised the question natural kinds.
OTOH I am heartened that people are effectively doing philosophy in the wild without getting too concerned with "whether other people have already said this before" which can be paralyzing.
I have the impression that people don't take philosophy very serious, and I would venture that that is because philosophy has such a meagre empirical record, nor a solid formal base outside of mathematics and logic. This makes it practically impossible to judge the merits of a particular idea.
A somewhat extreme example: does it matter what Heidegger has said about a certain topic for any discussion? Or Žižek, for that matter.
Philosophy provides rigorous arguments for and against various ideas, and assumptions behind those ideas. These ideas usually aren't empirical, but they are ideas many people have or questions people ask. Ethics, aesthetics, the nature of knowledge, truth, what exists, how to live, etc.
People who don't take philosophy very seriously still engage in those topics. They just ignore that the topics have a history of being discussed seriously and often put forth their own unexamined assumptions. Scientists aren't immune to this.
Žižek said some faintly reasonable things during his pandemic period, and semi-recently published an okay introduction to Hegel in the context of contemporary life.
Also the man is very hot when he takes his shirt off.
So they re-hash the same arguments with an equal lack of data and come to the same conclusions with equally dubious applications.
It’s not so bad when it’s just randos posting in internet forums, but now we’re seeing people with real influence leaping from unfounded premises to untestable conclusions with unswerving confidence.
What? Heidegger is one of the most influential philosphers of the 20th century. Dasein directly relates to the conversation about if an "intelligent" AI could exist and if it could/would have a sense of self similar to that of humans
Very briefly, the idea of a natural kind term is that of a term that “carves nature at its joints” (Plato’s metaphor), ie is not just a convenient way of splitting the world up into entities but one that reflects the true structure of reality.
Philosophers have debated various criteria for determining what constitutes a natural kind, which I think would have added some clarity to the article.
A related approach that could have added clarity (though I can’t recall whether it deals with natural kinds per se) is Hacking’s Social Construction of What? which lays out a sort of hierarchy for how socially constructed a given phenomenon is. It attempts to add rigor to our intuitions around how objective or subjective a given judgement is.
I would word your statement a little differently. What you said sounds very confusing to me, as if you are conflating intelligence with a lot of other things.
Intelligence is discernment, heuristics and pattern recognition.
Agency is achieving goals.
Wisdom is the process of discovering which goals are worth achieving.
It's pointing out a very important, uh, point. Yeah, it's creepy to make people just to enslave them (Asimov's Three Laws of Robotics are also the three laws of a perfect slave, so to speak.)
And, yes, it's incredibly stupid to build slaves powerful enough to turn you out (Asimov has a few stories that touch on that, IIRC.)
> There have already been intelligence explosions... reading and writing... iodine supplementation...
It's dubious to refer to these individual events as "explosions". They're just single significant changes. Any event in humans or AI you could respectably call an explosion would be a cascade (note: causally interconnected) of a large number of these individual intelligence-increasing changes.
If anything, I'd argue that the buildup of communal knowledge in human civilizations over the last several thousand years, especially the last few centuries, has been an explosion of sorts. It's been a huge increase in capability for one species, and would look like a step change on a timescale that includes all Earth life. But if that's an explosion, writing was just one of the chemical reactions.
The trick behind Alexander’s supposedly non-platonic correlationism is that the hierarchy of “sub-claims and sub-sub-claims” are difficult to enumerate for specific cases. Because of this it’s not immediately apparent that the bundle of correlated subclaims are filtered for relevance. All the Platonism is hiding under that rug of relevance.
That’s to say, when we talk about Tyson being stronger than my grandmother, we ignore correlated subclaims such as “Tyson consumes more protein than my grandmother” or “Tyson wears a jockstrap more often than my grandmother”—these are correlated with being stronger (probably), but they are incidental to the concept of strength. When we ask ourselves what subclaims are relevant, we’re right back to pointing at essence.
And correlations of what? Very strong emphasis on what. Quid est?
From the get go, like all skepticism, this approach begins by sitting on a branch that more or less acknowledges reality and its comprehensibility, but quickly begins sawing off the very branch it is sitting on. Skepticism is always a form of selective denial, and as such, it is incoherent. It's like those people who argue passionately that there is no truth while in the lecture hall, but as soon as the clock strikes 4pm, they're worried about missing the bus and getting everybody in the room to the next lecture, or as Duhem would say, going home to kraut and pipe.
"Triangularity" is not a "bundle of correlations". Triangularity is an irreducible, but analyzable whole, a form that is instantiated in the world within particulars and that we encounter in particulars and abstract from those particulars to arrive at the concept. That's what a concept is: a form abstracted from concrete particulars that exists in the intellect as a universal.
Unfortunately, mechanistic metaphysics, taught and insinuated in our scientistic curricula from a young age, is a difficult intellectual habit to break for many. It is very difficult for people to uproot these obstinate presuppositions.
Intelligence, whether you use it as a fuzzy concept or a narrow one, assumes a goal. It's impossible to say if an actor acts intelligently, without assuming what it's trying to achieve.
It's peak this guy, and peak "rationalists" in general, to defend the concept of "objective" intelligence, intelligence towards unspecified goals. This is what they actually care about: that there's an objective scale of intelligence, and that they're above most people on it. Everything else they believe is tangential. They choose other beliefs according to how they can strengthen the core belief, that they are the intelligent ones. (And you bet they downvote people challenging it).
What goals their intelligence are directed at, they're evasive and inconsistent about, and that others might have different goals is certainly not a permissible option. In THAT area, they are platonists, no matter how fuzzy their forms are: playing the flute is the task of a professional flautist, and steering the ship of humanity is a task for professional captain smart. Where the ship should be heading is not a subject of debate.
If a process has in it a place where a representation of a goal is stored, and for a wide range of goals, if represented in this format, if stored in this location, the process would tend to effectively achieve the goal, then it seems like the “what the goal is” (within the aforementioned wide (but not necessarily universal) range of goals) and the “capability of achieving goals” (within that range of goals) would be things that could be factored out as separate properties.
Do you think that that condition doesn’t occur?
Or, do you think that the “the range of goals for which this works” being limited, makes the “how capable is this process at achieving goals” concept illegitimate, no matter how broad the limited range is?
Also, the reason you are getting downvoted is presumably in part because you talked about being downvoted.
Your accusations are false and uninteresting, so I’m only addressing the object-level topics I can extract from your comment.
You can't pursue all goals equally well, unless you don't pursue goals at all. So when you start talking about "goal representations", you implicitly talk about a set of possible sub-goals - the set you would like to be able to represent, and the process by which you plan to pick them, implicitly defines the actual goal.
You're not interesting either, "rationalist", and I think you believe a lot of false things too. But unfortunately people who should know better listen to you.
I said the accusations weren't interesting, not that you weren't interesting. I'm sure you have plenty of interesting things to say on other topics, and the other parts of what you are saying on this topic are (clearly) interesting enough to engage with. I'm just not interested in mud-slinging.
I'm not entirely sure what it would mean to say that something is equally competent at pursuing any possible goal, but it does seem very likely to me that, whatever it might mean, things can't be equally competent at all possible goals (for reasons of description length, if nothing else).
Ah, hm, it seems you are phrasing things in terms of simultaneously pursuing all possible goals?
I think that's a rather different thing than what I'm describing. Do you agree?
Something can be highly capable with regards to each of two goals which are each-other's opposite (while perhaps not in fact pursuing either), but it certainly can't be simultaneously be effective in pursuing two goals that are in direct opposition.
[emphasis]
None of these things are obstacles to it being possible to say of two mechanisms W and Z, that for every goal, that if W has any competency towards achieving that goal at all, then Z is at least as competent towards achieving that goal.
[/emphasis]
(where what it means for W or Z to have competency towards a goal, is that, the goal can be encoded in whatever goal-representation system the mechanism uses, and if that representation were to be what is stored in the part of the mechanism that determines what goal is pursued, then the mechanism would be effective at achieving that goal.
Though I suppose in some cases it might be unclear whether two representations in different systems are representing "the same goal"?)
You say that talking about "goal representations" implicitly talks about a set of possible sub-goals. I guess you mean, like, goals that would be used to achieve some larger fixed goal? I'm not convinced that has to be true. It could be that the end-goal is exactly the one which is represented.
Additionally, I don't see a reason why there can't be any languages capable of representing arbitrary goals.
At the least, it seems pretty clear to me that there is a language capable of describing any particular goal you could communicate to me, and so any refutation that you might provide me, of the claim that there are such universal goal description languages, would have to be non-constructive, and this makes me doubt the relevance of any such counterexamples.
You need to be really careful about not assuming what you're trying to show when talking about teleology. I know teleology is something like a swear word in old new-atheist circles, but when we're talking about goals, it's teleology as literally as it gets.
You brought up goal representations. Can you explain what a representation is, without sneaking in the notion of "meaning", and thus the notion of "goal"? I certainly can't!
I’m not an atheist. I’m a Christian. I don’t have an objection to bringing up teleology.
Are you asking if I can, (without sneaking in a reference to meaning) explain what a representation of something is, or specifically what a representation of a goal is?
It seems possible that my choice of the word “representation” gave a different impression than I intended. I meant basically the same thing as “encoding”. If that’s the meaning you got from it, then cool, my word choice didn’t cause a miscommunication.
If I have a computable function from finite bit strings to Turing machines, this defines an encoding scheme for Turing machines. A system of representations of Turing machines.
Is that “sneaking in a reference to meaning”? In a sense that implies a notion of “goal”? If so, not in a way that I think matters to the point.
Perhaps one could say that, by describing it as being an encoding scheme of Turing machines, that I am therefore saying it is an encoding scheme for Turing machines, as in, with the purpose/goal of specifying a Turing machine. This, I think has some truth to it, but it doesn’t imply that some artifact which relates to such an encoding scheme in some way, has that as its goal, so much as, me describing something in terms of an encoding of Turing machines, says something about my goals. Namely, I want to talk about the thing in a way relating to Turing machines.
If what you were challenging me to define without a hidden reference to meaning/purpose was specifically a system of representations of goals, then,
well, if by “meaning” you just mean like, “statements and predicates and such”, then, I would say that defining what it means for something to be a scheme for representing goals, should, yes, require referring to something like a correspondence between (encodings/representations) and, something like predicates or conditions about the world or orderings on potential configurations of the world or something like this. Which, in some sense of “meaning”, would, I think, include at least an implicit reference to that sense of “meaning”.
So, if that’s your claim, then I would agree?
But, I don’t think that would imply much about the overall goal of a system.
If I purchase a DVD player which has as parts various microcontrollers which technically are capable of general computation, just because it has some processors in it that could hypothetically execute general-purpose programs, doesn’t prevent the overall purpose of the DVD player from being “to play DVDs”.
Of course, there’s a (probably big) difference between “encoding a program” and “encoding a goal”.
But, in the same way that a device can have components capable of general computation, if only the program were swapped out, without the use of the device intended by the manufacturers being “do general computation”,
I would think that a system could be such that, considered in terms of a particular encoding scheme for goals, if some part of it which (viewed in terms of that scheme) stores some encoding of some goal, and if modified to have an encoding of a variety of other goals (viewed in terms of the encoding scheme) would result in the goal being furthered by the system,
That doesn’t imply that the goal the system was designed to achieve, nor the goal which it currently pursues, is “be able to pursue any goal in this encoding scheme”?
(... wow I phrased that badly... I should edit the wording of this, but going to post it as is first because phone low on power.)
It seems fairly likely to me at this point that I’ve misunderstood you? If you notice a way I’ve likely misunderstood, please point it out.
Tell me you've just read a philosophy Wikipedia article for 5 minutes without telling me you've just read a philosophy Wikipedia article for 5 minutes. This is not what platonism and nominalism means lol
Because animal intelligence (including human intelligence) is a blob of neural network arranged in a mildly clever way that lets it learn whatever it wants efficiently.
"Variation in IQ during adulthood is about 70% genetic."
The proviso "during adulthood" is important: "The heritability of IQ increases with the child's age and reaches a plateau at 18–20 years old, continuing at that level well into adulthood." (Is it just me or is that a weird thing for it to do.) That being said, the Wilson Effect paper goes on to say, "It is important to specify the populations to which any results can be generalized and not misinterpret what they mean. The samples were drawn almost exclusively from Western industrial democracies. These settings have characteristic environments. Only a few of the participants were raised in real poverty or by illiterate parents, and all study participants had access to the contemporary educational programs typical of those societies. This is the domain to which we can generalize." In any case, for most of my life the variation in earring-wearing was almost 100% genetic, so there's that.
"A lot of this seems to have to do with literal brain size (which is correlated with intelligence at about 0.2)..."
"There have already been intelligence explosions. Long ago, humans got smart enough to invent reading and writing, which let them write down training data for each other and become even smarter (in a practical sense; this might or might not have raised their literal IQ, depending on how you think about the Flynn Effect)."
This is a direct contradiction to Alexander's apparent argument above, unless you consider reading and writing to have a significant genetic component. (Heritability? Oh, yeah, I'd expect reading and writing to have pretty high heritability, at least historically.)
But is knowing things really what you mean by intelligence?
"Later on, we invented iodine supplementation, which let us treat goiter and gain a few IQ points on a population-wide level."
Fortunately, iodine supplementation is also highly heritable.
"...were emphasizing how rather than magical intelligence, AI would need slow, boring work on the nitty gritty of computation..."
How are LLMs doing on that whole hallucination thing? The ability to solve novel problems?
This article segues from the failure of 80s style expert systems (in many domains) to a generic argument for x-risk and superintelligence explosions. But it doesn't address any of the obvious counter-arguments except one about the definition of intelligence, I guess because Alexander feels they're already addressed.
Once you get smart enough, you can do things that make you even smarter. AI will be one of those things. We already know that bigger blobs of compute with more training data can do more things in correlated ways - frogs are outclassed by cows, chimps, and humans; village idiots are outclassed by Einstein; GPT-2 is outclassed by GPT-4. At some point we might get a blob which is better than humans at designing chips, and then we can make even bigger blobs of compute, even faster than before.
But what about:
1. Limited to human performance by training data? OpenAI apparently aren't training GPT-5 because they think that research direction is tapped out. Their focus has been on augmenting this "superintelligence" with boring logic-based systems like calculators, Python interpreters, web browsers and 80s style expert systems like Wolfram Alpha. All this is suspiciously like what a human would need, not a superintelligence. It implies they don't think they can do another 2->3->4 style leap, probably due to lack of training data that would yield more advanced capabilities.
2. Bottlenecked by physical experimentation? Alexander casually asserts that if you trained an AI on circuit design it'd immediately do better, and that'd be used to build better AI chips, which in turn would yield a smarter AI ad infinitum. But Google already tried this and it just led to fraud claims, not better chips. And even if an LLM came up with an idea for a better chip, humans would still need to do the physical experimentation to figure out how to build them.
3. Bottlenecked by lack of imagination? LLMs can be "creative" in the artistic sense but there is a suspicious and very noticeable absence of them coming up with any genuinely interesting ideas. They can think faster than humans, and know immeasurably more, yet has anyone found even just one example of the sort of out-of-the-box thinking that would be required for AI to outsmart humans? Where are all these scientific breakthroughs the AI evangelists keep promising? GPT-4 is pretty damn smart but despite asking it many things, it never once came up with an idea I hadn't already had.
The article's examples seem to be its own undoing. There is no such thing as strength-leading-to-more-strength in some sort of recursive loop, which is why he needs such a bizarre and artificial example. Why should we believe there is for intelligence, when the history of human intelligence is a 2000 year struggle for even quite minor improvements in cognitive ability and even that is highly debatable?
I think it's an important unanswered question whether intelligence really is the bottleneck for many of the things that require it. Is the production of better and better computer chips really bottlenecked by the intellect of the designers, or by simulation software and the back and forth between design and the physical process of prototyping and testing?
And if intelligence is not the bottleneck... well... is superintelligence actually worth as much as we think it is? Is human intellect the apex of what biological systems can do, or is it merely the point past which intelligence stops being the bottleneck and the returns of higher intelligence drop off dramatically?
Yes, fully agree. There are a lot of unstated but unintuitive assumptions and intuitions going on in the AI risk/ethics community. It's useful to surface those.
The argument made in the article is that “Is Albert Einstein smarter than a toddler?” means things like "Einstein can do arithmetic better than the toddler".
But this only makes sense because the toddler and Einstein are similar systems doing the same fundamental things when they do arithmetic. We would accept that "Einstein plays chess better than a toddler" is an indication that Einstein is more intelligent, but would we accept the same from Deep Blue? If not, why not?
The whole point is that intelligence is a concept that points at a bundle of related characteristics.
There is one indication that Deep blue is more intelligent than a toddler - it can beat the toddler at a game of chess.
There are other indications that a toddler is more intelligent than deep blue - they could write better poetry, play tic-tac-toe better, play checkers better, play, well, literally _any_ other game better, do arithmetic better, etc.
So on one metric DB is more intelligent. On every other metric the toddler is more intelligent.
Is it actually an indication that Deep Blue is more intelligent, or does it turn out that Deep Blue is succeeding at chess by means other than intelligence?
In which Scott demonstrates that he has no idea how "strength" works, or frankly what it even is.
His quality has been sliding the last couple years, which happens to coincide with his announced commitment to AI Doom Ideology. Maybe those are connected and maybe not, but there it is.
He's not sliding. He's been down there at the bottom all along. Scott Alexander is fond of numerous neoreactionary ideas, especially those around race and IQ (so-called "human biodiversity"). He's admitted it himself. In his own words:
"I am monitoring Reactionaries to try to take advantage of their insight and learn from them. I am also strongly criticizing Reactionaries for several reasons.
"First is a purely selfish reason - my blog gets about 5x more hits and new followers when I write about Reaction or gender than it does when I write about anything else, and writing about gender is horrible. Blog followers are useful to me because they expand my ability to spread important ideas and network with important people.
"Second is goodwill to the Reactionary community. I want to improve their thinking so that they become stronger and keep what is correct while throwing out the garbage. A reactionary movement that kept the high intellectual standard (which you seem to admit they have), the correct criticisms of class and of social justice, and few other things while dropping the monarchy-talk and the cathedral-talk and the traditional gender-talk and the feudalism-talk - would be really useful people to have around. So I criticize the monarchy-talk etc, and this seems to be working - as far as I can tell a lot of Reactionaries have quietly started talking about monarchy and feudalism a lot less (still haven't gotten many results about the Cathedral or traditional gender).
"Third is that I want to spread the good parts of Reactionary thought. Becoming a Reactionary would both be stupid and decrease my ability to spread things to non-Reactionary readers. Criticizing the stupid parts of Reaction while also mentioning my appreciation for the good parts of their thought seems like the optimal way to inform people of them. And in fact I think it's possible (though I can't prove) that my FAQ inspired some of the recent media interest in Reactionaries."
Einstein wasn't (not is) more intelligent than a toddler in the same way he was stronger than a toddler, or maybe at all. He certainly knew more. He was actually less intelligent than he, himself, was as a toddler if one is measuring at rate of knowledge acquisition rather than acquired.
Once the analogical reasoning broke down I stopped reading, especially since I believe life is a prerequisite for intelligence and is bound to it, because intelligence does not equal computation and is more than Turing complete.
The main one is that I’ve never heard anyone describe in any detail how the technology will work. I’ve worked in big shops running big online ML systems and oh boy, do they need a lot of diaper-changing to even stay running. If all the ops folks went on vacation it’s a “hours vs days” not “weeks vs months” question of how soon it would fall over. Some log rotate thing gets stuck, and away we go with cascading failures.
So how, in some detail, do we get from big-ass mixture model of transformers to even self-operation? That’s got to be a pre-requisite for self-improvement right? inb4 “The risk is so great the details don’t matter because not impossible”, hmmm, no. Extreme tail risk isn’t interesting when addressing it comes at the cost of driving immediate, overwhelmingly likely risk of nightmare outcomes through the roof. Consolidating the alignment and steering of big models into the hands of a few CEOs, governments captured by them, or both is a clear and present danger of horrifying proportions. I take mitigate that over extreme tail risk sure as Tuesday and taxes.
Then there’s the reality of AI development thus far, which is that it comes in stops and starts. Clearly the past is no guarantee of the future, but it seems a damned sight better as a prior than the log scale and a ruler methodology employed by Yud or whoever.
Smart people talk like this is serious. There are even some elite practitioners who talk kinda like this (Hinton, Karpathy on Lex, some others). And I don’t want to be walking around with my head up my ass on it, so I’ll be grateful to anyone who wants to set me straight.
But if there’s an explanation somewhere, by a credible practitioner, of how this could actually happen in some actual technical detail, I haven’t found it.
I think that needs to be in the footnotes the next time the mainstream press starts publishing blog posts off LessWrong as consensus.