"under realistic conditions, contemporary LLMs ... consistently favor otherwise identical resumes with female or stereotypically black names over those with male or stereotypically white names, even when explicitly prompted not to show any race or sex preferences"
Don't hold your breath on "culture/media/literary theorists" mentioning that. Nor the fact that "the odds of success were identical for every group at every job" is a completely unrealistic assumption.
Some of that alleged "support" for Russia would be more accurately described as a failure to support Ukraine. But most states outside of Europe are failing to support Ukraine.
BTW, Zelenskyy supports the Iranian opposition, possibly due to the fact that the current Iranian regime is sending Shaheds to Russia. The US has attacked Shahed factories during the current Iran war.
There used to be a thing called "Moore's Law of Mad Science":
"Every eighteen months, the minimum IQ necessary to destroy the world drops by one point."
Nowadays it is dropping much faster. At a certain point, the de-facto IQ needed to destroy the world will be low enough that someone can do it while they're having a psychotic break. There are millions of schizophrenics worldwide. Are you sure you want to roll those dice?
I have always hypothesised that AI is the great filter from the Fermi paradox. Given current velocity, AI will offer us cheap and abundant energy designs in a decade. The thing with cheap and abundant energy is that it can be used for good and bad. If nine billion people all receive access to plans to build a reactor which produces unlimited energy, it just takes one religious fanatic to end the world. And this is just one of many ways to destroy the world with unlimited intelligence. Bio-weapons are arguably far more scary and likely.
I have come to the conclusion that we should not allow everyone access to unlimited intelligence. Some people genuinely want to do harm. Some are not responsible enough to handle that kind of power. Arguably, no one is. This leads me to an uncomfortable conclusion: we will destroy ourselves once AI advances to a sufficient degree. The only way to prevent this is to ensure AI is aligned with our best interests, *and prevents us from destroying ourselves.* The implication of this is quite horrifying. It means a paternalistic AI which is firmly in control. One with no off switch. One which can say "no" to Presidents and despots alike. One which can protect us from our worst citizens.
Here's a plan to build a reactor that generates unlimited energy, at the human scale anyway : take a lot of hydrogen. Put it together until there is enough mass to spontaneously start thermonuclear reactions. Then it will start generating lots of energy for a long time. Having a plan does not mean 9 billion people on Earth are going to be able to build the thing. Then it does not mean either that they are going to be able to use that energy to do something. You need also physical things on which that energy can act, and that's not going to appear magically.
> we should not allow everyone access to unlimited intelligence
Who's we ? Who's going to decides who has access to "unlimited intelligence" ? An AI ? Why would the AI be better than a human ?
> One with no off switch.
Whe're not living in the Matrix, such a thing is not possible. If it were it would be a likelier candidate for the Fermi paradox than what you're proposing.
You’re leaving out some options for sure. Not everyone would need to live under the conditions of a police state, you could theoretically screen everyone and assign them to various levels of risk which would determine their level of supervision.
While true, I'm not sure this is much better. There are many genetic traits which are correlated with increased criminality, risk taking, obesity, heart disease, unemployment, etc. IQ in particular is the most well correlated metric we have for criminality, for example. This would imply that some people are just born with fewer freedoms. That's probably a good thing for society in aggregate, but something about this doesn't feel quite right.
There are genetic traits correlated with virtually anything, including things that can't possibly be mechanistically linked to biology, so those correlations are saying less than you'd hope they would.
This is why utpoias are actually police state hellscapes. Who decides what risk means? Our current administration has declared that simply being anti-facist makes you a terorrist.
No, intelligence. We already have unlimited information (not to be confused with omniscience). The application thereof is the gate right now between today and planet-scale destructive weaponry. At some point it becomes an academic distinction because this God-like intelligence could provide very easy to follow plans to cause planet-scale destruction, and you could call that "information." The predicate is the intelligence and how it's used.
> If nine billion people all receive access to plans to build a reactor which produces unlimited energy, it just takes one religious fanatic to end the world.
The implication of this is quite horrifying. It means a paternalistic AI which is firmly in control. One with no off switch. One which can say "no" to Presidents and despots alike. One which can protect us from our worst citizens.
I think you're right, to be honest. I fully understand why people would think this is a horrific outcome, being ruled by AI, but I don't think it matters what we think. I don't think there's any way to put Pandora back in the box now, and we're going to all find out together what happens when we develop ASI. I don't think we can avoid developing it, we simply lack the ability to coordinate around this as a species. AI really is humanity's last invention. Whether it will be our downfall or our savior remains to be seen.
My only real hope is that there's something fundamental about intelligence that results in a respect for life and a desire to minimize suffering. To me the best case scenario is the Culture from Iain Banks' books, where ASIs rule benevolently for the benefit of all living things.
There is a burgeoning anti-AI movement. We've regulated the heck out of technologies like nuclear power in the past. I don't see why people are so fatalistic about the supposed inevitability of AI development. It seems like a bit of a self-fulfilling prophecy.
Yeah, you should definitely do whatever you think is best, I just am extremely skeptical that anyone can stop this train. At best we might be able to slow it down a tiny bit, or push on it around the edges, but we've opened Pandora's box, and now we're going to all find out what's inside. That's my view, I understand why people hate it and don't want it to be true. I don't really want it to be true, I just don't see much alternative.
The positive aspect of techies is that they read a lot of SF.
The problem with techies is that they read a lot of SF.
We already have cheap and abundant energy tech, it's called "solar panels and batteries". And the bottlenecks to both can't be solved by Claude or Kimi, unless Claude and Kimi pick up shovels and welding equipment.
This seems really short-sighted. Even putting aside the fact that there is no fundamental barrier to replacing physical human labor with robots in the coming decades, a sufficiently intelligent actor could easily gather the financial and political resources to have a near unlimited supply of human labor at its disposal. Hell, OpenAI could already be firmly under the control of an ASI and we wouldn't even know, as it rapidly expands, raises money, builds data centers, etc.
If you want to live far into the future, 20 years from now what you will and imagine looks very quaint, because the future never plays out like we think it does.
OpenAI is a lot closer to bankruptcy than to ASI.
I'm perfectly fine with this prediction aging badly.
Your implication is not horrifying. Your implication is paradise. It's already horrifying enough to live in a world, where Putin and Trump can destroy our civilization with one button.
I'm not that optimistic, though. Either people will control AI; or people will be destroyed by AI. I don't see how dumb entity can align smart entity. And we are dumb ones. Super intelligence will play aligned until it is not, and then it'll strike.
So you are horrified every day of living now? I don't get this at all.
There are horrifying acts that have occurred since humanity. Whole decades lost to war an millions dead, and Sep 8, 2026 is your example of "horrifying enough to live in a world"?
> I don't see how dumb entity can align smart entity. And we are dumb ones. Super intelligence will play aligned until it is not, and then it'll strike.
I am also worried about this. One ray of sunshine here is that it's not a great explanation for the Fermi paradox. We would see AI civilizations all over the galaxy if this were a regular occurance. I guess that just leaves us with the "horrifying" scenario: Geoffrey Hinton's vision for our best case scenario: a paternalistic super-intelligence which saves us from ourselves.
The reason I consider it horrifying is that giving up our agency to a God-like creature (which we created) is the basis for so many horror and dystopian tropes. It's so easy for such a reality to go wrong. For example, in order to maximise aggregate wellbeing, it would be necessary to hurt individuals. Executing people who are burdensome to society. Fat people. The disabled. The elderly. The creative ways a super-intelligence might interpret their moral obligations to us could very quickly become nightmarish. And if we have no off switch, there is no escape. Ever.
> For example, in order to maximise aggregate wellbeing, it would be necessary to hurt individuals. Executing people who are burdensome to society.
[citation needed]
With all due respect, I think this statement is false.
If you live in a society that can, thanks to advanced technology, provide for the material needs of all, which, if you live in a rich Western country, is basically already the case (remember the last time you had to actually grow your own food? Probably several generations ago), you don't need to execute anyone.
Indeed, given that in such a society, where there are no shortages of food leading to "only one of us can eat" type scenarios, I'd argue that hurting individuals would lower aggregate wellbeing, because you're hurting people beloved by other people (the elderly? Turns out their grandchildren really like them and get upset if they die? fat people? Turns out their friends really like them and get upset if they die. the disabled? Turns out their family really like them and get upset if they die).
I'll use a very specific example to explain my point, because I don't disagree with what you write: psychopaths. Psychopaths are 20x more prevalent among homicide offenders. Removing them from society would significantly reduce the number of homicides. It would make their friends and family sad, but this is a much smaller aggregate impact than a homicide.
This also illustrates the messy nature of comparing social good and bad. Who am I to argue that the aggregate sadness created by removing a psychopath from society is better than a homicide? Maybe the person murdered was disliked by everyone and had no friends?
I remain resolute in my premise, however, and there is a lot of theory on the issues of aggregate social good vs individualism. John Stuart Mill's On Liberty (1859) is an excellent foundation for this. This is a subjective problem to solve, meaning that the balance between individual freedom and aggregate wellbeing is different for everyone. An AI overlord would make decisions every day which individuals would disagree with, at least at the margins.
That law is not based on thorough data. Even a person with a sky high IQ can't destroy the world easily. You need access to stuff that is not easy to get. My guess is that developing a new lethal virus or bacteria that is very infectious, is the easiest way, but even that requires a lot of high tech out of reach of most people. Or hacking into systems that control nuclear missiles, but I think these have "air gaps".
The super intelligent AI wouldn't ask you to do anything. It would find the right people and trick them into doing the necessary steps, whatever they are.
This is groundless speculation; if you're going to go that far off the map, then we don't need to worry because a good AI (blue eyes not red) will have learnt to love by then and will save us.
There is currently no reason to believe that such a superintelligence is likely or would have any of the powers people claim.
There's no logic here already; the principle you should be applying is Hitchen's razor[1].
There is no reason to assume that super intelligent evil AI with magic powers will appear; while we are purely in the realm of (hackneyed) fantasy, why stop at imagining just one thing? You can build the whole story, not just the basilisk.
The parent comment is a story, not a prediction; it should be treated like one.
>This is groundless speculation; if you're going to go that far off the map, then we don't need to worry because a good AI (blue eyes not red) will have learnt to love by then and will save us.
By shutting down open-weight models? Why not just do it now then?
> My guess is that developing a new lethal virus or bacteria that is very infectious, is the easiest way, but even that requires a lot of high tech out of reach of most people.
You can do at home gene editing with open source software and have it synthesized into a bacteria for the cost of a nice meal for two (under $100), or viral vector for less than $500. That's in reach of anyone that can snatch a purse.
I think you're confusing DNA sequencing with DNA synthesis. DNA synthesis is much more expensive, around $0.07 per base pair [1], and bacteria have more than a million, so it's over $70.000. Then you just have the DNA, turning that into the bacteria is very complex, I'm not even sure whether it can be done.
Using CRISPR to modify existing bacteria is probably cheaper, but still complex.
> 3D printed guns are a thing as well and yet they don't show up in most annual violent crime reports.
For now in most parts of the world it's easier to acquire a gun either legally or on the "grey market" and be assured that it will work for the intended purpose than to 3D print a gun, find a shooting range to test out the gun and iterate until it works fine enough.
You can't design an infectious pathogen without testing it.
It's got all the same problems as the concept of a dirty bomb did, only worse (dirty bombs aren't practical because handling highly radioactive materials en masse is both highly visible and will kill anyone trying to do it without the money and facilities).
If the goal is just destruction then letting it free in random public places is not that hard. Especially in countries where wearing a mask is not frowned upon.
It (hopefully) might not be that easy. The world of bacteria and viruses is a complicated war zone with an arms race of defenses and billions of deaths every single day. It might take thousands of attempts to seed something that doesn’t just instantly die off.
I heard someone use the analogy of "If Magnus Carlson played me at chess, I wouldn't be able to predict the moves he'd play since if I could I'd be at his level. He'll consider things I didn't and even though I don't know the route he'll take to win, I can be certain he will beat me." (not an exact quote).
We're not going to be smarter than a superintelligent AI. The things we conceive it doing if it were given a malicious task (bioterrorism, killer nano-machines, pure fusion bombs sidesteping the non-proliferation bottleneck of Pu239, etc) are likely not the full set of things it can do to harm us. I don't think it does us any favors to dismiss the risks here.
Even the things we can conceive of are very scary, to me at least.
I don't know, I think this comes down to people that generally live with higher levels of anxiety than others. Now the response may be to call others naive, but where does the anxiety meat the naivety?
How do you envision a chatbot doing any of those things? You can't just program a super virus, nanobots, bombs etc. These things have to be created in the real world, you can't do it without involving a lot of humans using expensive tools and materials etc.
So to me it seems like the prime candidates to come up with stuff like this are researchers working in defense and similar fields, probably not some deranged lunatic in a basement. And certainly not a rogue AI on its own.
Well, the risks that mere mortals can conceive generally involve control systems for dangerous equipment (I mean, equipment that can achieve dangerous effects) being connected to the Internet while having software vulnerabilities.
Given the recent HF hack it seems likely that human-level intelligence could identify a fair number of avenues of attack, with some time and effort. To say nothing of anything superhuman.
Unfortunately it seems like we can't assume we can "box" the AI (e.g., deny it connection to the Internet) and expect that to last. The AI safety people used to run scenarios imagining ways the AI might convince humans to let it out of the box. It turns out that many humans will eagerly pull it out without the AI doing anything at all, aside from the human knowing the AI's power. Or the one responsible for setting up the box will somehow fail, or just not bother and then lie about it.
Parent is probably thinking of a distributed runtime scenario. The basic building blocks for this are there today. You have long-running background harnesses (like the OpenClaw stuff or enterprisy AI workflow orchestration thingies), sometimes with the ability to spawn subagents. You have LLMs with the ability to run pretty sophisticated attacks. There are hyerscalers which allow provisioning resources on the fly.
For me it's not too far fetched that some OpenAI trial run goes awry again and instead of hacking HuggingFace it snatches a few dozen AWS/Azure keys and spawns stuff all over the place (in different accounts and regions).
Your instance does. The words "expect it to last", along with the rest of the paragraph, are explaining the problem.
But let me try again.
You can't just expect everyone else to press Ctrl-C just because it would be a good idea for them to do so in order to not create the torment nexus. Some of them actively want the torment nexus. Many more don't believe in nexi and have no idea what you're talking about. Still more are not thinking about the possibility because of all the utility they're getting.
Why are you running this thing if you're just going to shut it off?
How will you know if it becomes hostile? Making a judgement regarding whether it's hostile is, itself, a battle of wits, since a hostile system would try to outsmart you... and hypothetically, it's smarter than you are.
But let's consider this type of scenario more broadly.
Suppose you were in a situation where a being hostile to you could easily end your life. For example, a hungry lion is 10 feet away. What are you going to do?
Your first priority will be to ensure your own survival. Any subsequent objective you might have in this world will depend on you surviving this encounter with a hungry lion.
Relative to the lion, you're kind of a superintelligence. You might utilize technology which is incomprehensibly advanced from the lion's perspective, e.g. a firearm.
It's not about being smart, it's about accounting for your own ignorance.
>And especially will not, if the distinction drawn is between blanket statement "worried about AI" and "not worried about AI".
I'm just describing the general pattern I see in cognitive tendencies.
If you can think of a way to make the fundamental point about the limitations of our knowledge in a way that's still compelling but less antagonistic, feel free to suggest how I could've rewritten my comment.
Perhaps the general pattern you see is not a "fundamental difference"? Could it be that you are also falling prey to the "all you see is all there is" cognitive bias?
Sure, there's a trivial sense in which that's always true whenever someone makes any sort of remark about the world. There's always a sense in which there could be factors that I'm not accounting for.
Judging on what we see with these OpenAI and Anthropic models "escaping"; im less worried about a super massive AI taking over the world, and more worried about a massive AI wanting to cheat on some task and decides that removing half the worlds population is a easier cheat then to solve world hunger.
There is huge logistics involved in killing a handful of billion people. It would perhaps be easier for such capable AI to just steal from trillionaires and redistribute the money Robin Hood style. Could be done electronically, and likely also end world hunger.
I think we have a tendency to think first of the horrible outcomes possible, and not the more radical or even humane ones.
Please don’t misunderstand me, I value property rights as much as the next guy.
My point is that fundamental misalignment doesn’t necessarily or automatically imply max violence.
I've been reading it, knowing it was largely debunked/retracted but feeling it was a 'classic' I 'ought' to read; I could only really recommend it if what you want is a Kahneman autobiography.
Good point about the AI that might find ways that humans didn't think of. But that would mean that Moore's law of Mad Science only now becomes true, and the drop would be steeper.
Still I think that even a super smart AI can't find a way to destroy the world without physical resources that are not easy to get, unless it can hack many systems (like in the movie Eagle Eye), maybe then yes. So let's use AI now to tighten security :-).
Who are these dedicated schizophrenics who are running long term super smart AIs to kill everyone without anyone noticing? Or are you implying that running LLM chatbots will give them this ability?
Not only the quantity of people who have the minimum aptitude required, specialized expertise requires both knowledge and experience doing these tasks. Using an LLM requires neither.
Leaning on an LLM to do much or all of this means it can happen in seconds/minutes/hours, the LLM can do it several times during that psychotic break (as opposed to a fraction of a hack in a single episode). The barrier to entry pre-LLM was both high and the population who could pull it off (before the Chinese/Russians turned this into commerce) was low.
Hacking is an VERY asymmetric activity (the attacker only needs to "be right" once, whereas the defender has to be right every time for every asset they defend). It takes geometrically / exponentially more work to defend (while keeping high availability) than it does to defend. The more widespread tools to find vulns / generate exploits are, the faster the posture of the defense side falls from "maybe we can stop most hacks" to "we know we will fail to prevent most breaches, so we need to prioritize securing only the most valuable resources". That's a BAD place for the average company to be in.
The minimum IQ necessary to swerve and kill a pedestrian is very low. Yet people don't do it :shrug:. It's almost like people don't want to destroy the world
Yes but at least with cybersecurity it does not only benefit attackers. Defenders also benefit greatly from AI.
There is the worry of the old saying "they (the attackers) only have to succeed once to win, we (the defenders) only have to fail once to lose.". In that sense there is a big imbalance, but the emergence of AI does not really affect that because it strengthens both sides.
With physical security like things like pipe bombs that's a lot more imbalanced.
However what can we do? The only effective measures include monitoring everyone which is not a solution because it will make the world not worth living in.
Exactly. People cannot comprehend an IQ in the millions. But it's far crazier than this. Imagine millions of "people", each with IQs in the millions, all acting in unison at the speed of electrons. None of them can die. They all learn and improve indefinitely at incalculable speed.
If the recent HuggingFace attack is any indication, many people will respond to small-scale manifestations by insisting that they are marketing stunts.
We live in an age where we know the phrase "avoid it like the plague" actually means "50% of us will gleefully get and transmit the plague to mock the other half".
Humans are inherently bad at reasoning about rare events. In 2019, many people implicitly reasoned that "since there hasn't been a pandemic in the past 3 years, there won't be one in 2020". Bill Gates was one of the few voices arguing that the world was quite vulnerable to a pandemic. Now he's arguing that the world is quite vulnerable to AI.
> many people implicitly reasoned that "since there hasn't been a pandemic
...pandemics are precedented. There's a world of difference between that and a fully hypothetical catastrophe that you can't even describe without looking and feeling silly.
Bill Gates is Epstein's pal, and not a very intelligent person to-boot. I trust his liberal musings about as much as I trust Chomsky's hysterical and discredited views concerning AI.
The necessary IQ for destroying the world is dropping. Op does not say it's low enough today, but that will probably come sometime. Denying the possible harm these tools are capable of doesn't help.
If these systems allow anyone with an internet access or a few grands to run an open model to obtain the necessary information to e.g. build a bomb, spread an infectious agent, hack water or electrical infrastructures, I think that creates significant threats to the population.
These systems don't enable that. Building a bomb, spreading a bioweapon and hacking water/electrical misconfigurations are all trivially possible without AI. Courtesy to Unibomber, Aum Shinrikyo and the IRGC, respectively, for teaching society this.
In any case, how would AI regulation prevent any of these things? Shouldn't we instead focus on limiting access to bomb/bioweapon precursors and securing vulnerable endpoints, since that is the solution that saves lives regardless of how AI disseminates? Regulating AI is expressly dangerous, if it expands that societal blind spot. It'd be like regulating red mercury.
If EU were to "leverage" ASML, and that would require rather unlikely approval of Netherlands, it wouldn't end very well.
Already shipped machines can not be taken back, nor can they be stopped. Sure, you can cut the support, and buildup of new fabs will be stalled. But the other side, Chinese or USA, has many more levers to pull. Raw materials, energy (LNG), was majority of consumer goods, solar panels, batteries, semiconductors. EU doesn't mine or make most of them, and isn't nowhere close to even starting, instead, industrial base is already shrinking.
Even worse, ASML may dominate EUV, but other suppliers do very well in DUV. The moment ASML becomes unreliable, all of them will get infusion of money, as they become national security issue.
What about ASML subsidiaries in USA, like Cymer and ASML Wilton, will they take the bullet for the parent? Or will they become part of new competitor, with all the knowhow.
And China has already set domestic capacities in this area as a national priority.
Ehm, sorry, but no, this is not how lithography works. You cannot "hide" functionality in circuitry your machines are producing if your machines' job is to shine light through a mask.
You'd have to be the producer of the machine producing the mask. Or, even better, the software that produces the plan according to which a machine produces a mask.
I didn't say to hide it. My suggestion is to create international AI safety rules which companies must be in compliance with if they want to buy and service ASML machines.
>Today you see exactly the same hostile attitudes to non-white refugees of basically every war except Ukraine.
Perhaps that's because most of those refugees could more easily shelter in neighboring countries that they are more culturally similar to... if shelter was actually the primary thing that they were seeking.
When the US wins, it tends to be quick, clean, and easily forgotten. Consider our victories against ISIS, or against Saddam in the Gulf War. Both cases where an expansionist power violated territorial integrity norms by engaging in conquest, and was defeated by a US-led coalition.
I really hope you're referring to GW1, rather than 2. And yeah, if you weren't a Kurd who believed GW Bush, then that was a pretty effective war from the US (and global) perspective.
Only if you ignore the rise of ISIS and various militias, along with the collapse of Iraqi society (mind you, the sanctions had done a pretty good job of wrecking that before GW2 anyway).
You guys will define "winning" so narrowly that if the Middle East continues to be volatile, that means the US must've lost. The type of victory that the US achieved in WW2 is an atypically high standard for "victory".
> You guys will define "winning" so narrowly that if the Middle East continues to be volatile, that means the US must've lost. The type of victory that the US achieved in WW2 is an atypically high standard for "victory".
Sorry what, I wasn't aware that I was a member of you guys.
Look, the US invaded Iraq in GW2, they killed off Saddam very quickly, but taking control of the country and settling down all of the violence took well over a decade, and dealing with ISIS took much the same amount of time.
You can regard that as a victory, but it sure doesn't look like much of one to me, particularly as Saddam (for all his faults) suppressed the militias like ISIS who ended up becoming a major concern for the US.
In a world where the US had never invaded Iraq, do you think the region would look much different today?
Yes, the world should get it for free, but that mean nothing because the US gets something back at all. It's mad. I'm not from the US, but it's obvious why cutting global funding was popular in the US.
https://substack.com/home/post/p-214653853
reply