Revisited this, and while it was tongue-in-cheek, it feels a bit ungrateful.
A lot of tech roles are intellectually stimulating and have a wage in the upper distribution of salaries. I think we have a very nice situation going for us.
Humans are not isolated nodes, we are more like a swarm, understanding reality via consensus.
The situation you described is possible, but would require something like a subverting effort of propaganda by the state.
Inferring truth about a social event in a social situation, for example, requires a nuanced set of thought processes and attention mechanisms.
If we had a swarm of LLMs collecting a variety of data from a variety of disparate sources, where the swarm communicates for consensus, it would be very hard to convince them that Moscow is in Connecticut.
Unfortunately we are still stuck in monolithic training run land.
> If you pretrained an LLM with data saying Moscow is the capital of Connecticut it would think that is true.
> Well so would a human!
But humans aren't static weights, we update continuously, and we arrive at consensus via communication as we all experience different perspectives. You can fool an entire group through propaganda, but there are boundless historical examples of information making its way in through human communication to overcome said propaganda.
The main reason for keeping AI static is to allow them to be certified or rolled back (and possibly that the companies can make more money selling fine tuning) — it's not an innate truth of the design or the maths.
While those are good reasons to keep the weights static from a business perspective, they are not the only reasons, especially when serving SOTA models at the scale of some of the major shops today.
Continual/online learning is still an area of active research.
We kinda do have LLMs in a swarm configuration though. Currently LLMs training data, which includes all of the non RAG facts they know, come from the swarm that is humans. As LLM outputs seep into the internet, older generations effectively start communicating with newer generations.
This last bit is not a great thing though, as LLMs don't have the direct experience needed to correct factual errors about the external world. Unfortunately we care about the external world, and want them to make accurate statements about it.
It would be possible for LLMs to see inconsistencies across or within sources, and try to resolve those. If perfect, then this would result in a self-consistent description of some world, it just wouldn't necessarily be ours.
I get where you are coming from, and it is definitely an interesting thought!
I do think it is an extremely inefficient way to have a swarm (e.g. across time through training data) and it would make more sense to solve the pretraining problem (to connect them to the external world as you pointed out) and actually have multiple LLMs in a swarm at the same time.
I was responding to the idea that an LLM would believe (regurgitate) untrue things if you pretrained them on untrue things. I wasn't making a claim about SOTA models with gigantic training corpora.
They have a dark pattern around annual subscriptions, i.e. I had a monthly subscription I wanted to downgrade, and with no confirmation they charged me for the entire year after I selected the lower tier and hit next.
I think the bitter lesson implies that if we could study/implement "how a machine with limitless cpu cycles would make our eyes see something we are currently thinking of" then it would likely lead to a better result than us using hominid heuristics to split things into sub-problems that we hand over to the machine.
The technology to probe brains and visual related neurons exists today. With limitless cpu cycles we would for sure be able to do make us see whatever we think about.
I'm not really familiar with that technology space, but if you take that as true, is your argument something like:
- We don't have limitless CPU cycles
- Thus we need to split things into sub-problems
If so that might still be amenable to the bitter lesson, where Sutton is saying human heuristics will always lose out to computational methods at scale.
Meaning something like:
- We split up the thought to vision problem into N sub-problems based on some heuristic.
- We develop a method which works with our CPU cycle constraint (it isn't some probe -> CPU interface). Perhaps it uses our voice or something as a proxy for our thoughts, and some composition of models.
Sutton would say:
Yeah that's fine, but if we had the limitless CPU cycles/adequate technology, the solution of probe -> CPU would be better than what we develop.
I think Sutton is right that if we had limitless cpu, any human split up would be inferior. So indeed since we are far away from limitless cpu, we divide and compose.
But i think we're onto something!
Voice to image indeed might give better results than text to image, since voice has some vibe to it (intonation, tone, color, stress on certain words, speed and probably even traits we don't know yet) that will color or even drastically influence the image output.
But if you assume that we have created something that is agentic and can reason much faster and more effectively than us, then us dying out seems very likely.
It will have goals different from ours, since it isn't us, and the idea that they will all be congruent with our homeostasis needs evidence.
If you simply assume:
1. it will have different goals (because it's not us)
2. it can achieve said goals despite our protests (it's smarter by assumption)
3. some goals will be in conflict with our homeostasis (we would share resources due to our shared location, Earth)
then we all die.
I just think this is silly because of the assumption that we can create some sort of ASI, not because of the syllogism that follows.
(As an intuition pump, we can hold on the order of ones of things in our working memory. Imagine facing a foe who can hold on the order of thousands of things when deciding in real time, or even millions.)
Point 1 is a big assumption. I am also not you, and although it's true that I have different goals, I share most of your human moral values and wish you no specific harm.
I'm also unconvinced by the idea that rapid reasoning can reach meaningful results, without a suitably rapid real world environment to play with. Imagine a human, or 8 billion humans if you like, cut off from external physical reality, like brains in jars, but with their lives extended for a really long time so that they can have a really good long think. Let them talk to one another for a thousand years, even, let them simulate things on computers, but don't allow them any contact with anything more physical. Do they emerge from this with a brilliant plan about what to do next? Do they create genius ideas appropriate for the year 3000? Or are they mostly just disoriented and weird?
I didn't mention it but I fully agree, I imagine ASI would have be to embodied.
My reasoning is simple, there are a whole class of problems that require embodiment, and I assume ASI would be able to solve those problems.
Regarding
> Point 1 is a big assumption. I am also not you, and although it's true that I have different goals, I share most of your human moral values and wish you no specific harm.
Yeah I also agree this a huge assumption. Why do I make that assumption? Well, to achieve cognition far beyond ours, they would have to be different from us by definition.
Maybe morals/virtues emerge as you become smarter, but I feel like that shouldn't be the null hypothesis here. This is entirely vibes based, I don't have a syllogism for it.
Smarts = ideas, and the available ideas are ours, which contain our values. Where's it going to learn its morality from?
* No values at all = no motivation to even move.
* Some ad hoc home-spun jungle morality like a feral child - in that case, it would lack heuristics in general and wouldn't be so smart. Even your artificial super-brain has to do the "standing on the shoulders of giants" thing.
* It gets its moral ideas from a malevolent alien or axe murderer - how come? Unless it was personally trained and nurtured by Mark Zuckerberg I don't see why this would happen.
Mind you, I suppose it's true that even normal humans tend to be somewhat mean and aloof to any outgroup of slightly different humans. So even after it learns our values, there's a definite risk of it being snotty to us.
The "different goals than us" part is redundant. Across humanity there is already vastly oppositional goals. That's why we have prisons, war, and counter-terrorist organizations.
As Geoffrey Hinton points out, a generally useful subgoal of any task is power accumulation. In other words, you can assume that a very intelligent AI will always be not just smarter than us but also accumulate power for anything that you ask it to do, simply in order to do that thing more effectively.
Imagine if everyone had access to a magic genie. Eventually someone is going to wish for something bad.
Where I think Hinton’s views fall down is that the e have zero idea of what AGI smarter than us might want or what it might do. Like people always talk about it as if an entity like that would just hang around and bully our species. It might evolve into a ball of light and leave the planet. I don’t know but we seem to assign a lot of human traits to something that wild likely be completely unrecognisable to us in probably twenty minutes after birth.
What's the difference between desires and goals in this context really? You could say he is worried about a reasoning machine "relentlessly programmed" to achieve some goal, but a reasoning machine might just reason itself out of everything you've told it to do. Something so creative, so capable, so beyond us, yet it's going to...assassinate other people for you? Why?
When something goes from being a computer program to a self-aware, conscious being with agency, things change a lot.
Hinton is a major paradox of a human, he has spent his life building the very thing he says will likely doom us, and now spends his life warning us against his own work? So much of this AI doomerism just seems like a "chinese finger trap" for the ultra logical thinker.
It's a fucking weird time to be alive. The 90s felt much less weird and dystopian to me.
- regurgitate entire passages word for word, until that behavior is publicized and quickly RLHF'd away
- rip github repos almost entirely (some new Sonnet 3.5 demos Anthropic employees were bragging about on Twitter were basically 1:1 to a person's public repo)
It seems clear to me that not only can copyrighted work be retained and returned in near entirety by the architectures that undergird current frontier models, but the engineers working on these models will readily confuse a model regurgitating work to be "creating novel work".
> We need systems that try to be coherent, not systems that try to be unequivocally right, which wouldn't be possible.
The fact that it isn't possible to be right about 100% of things doesn't mean that you shouldn't try to be right.
Humans generally try to be right, these models don't, that is a massive difference you can't ignore. The fact that humans often fails to be right doesn't mean that these models shouldn't even try to be right.
By their nature, the models don’t ‘try’ to do anything at all—they’re just weights applied during inference, and the semantic features that are most prevalent in the training set will be most likely to be asserted as truth.
They are trained to predict next word that is similar to the text they have seen, I call that what they "try" to do here. A chess AI tries to win since that is what it was encouraged to do during training, current LLM try to predict the next word since that is what they are trained to do, there is nothing wrong using that word.
This is an accurate usage of try, ML models at their core tries to maximize a score, so what that score represents is what they try to do. And there is no concept of truth in LLM training, just sequences of words, they have no score for true or false.
Edit: Humans are punished as kids for being wrong all throughout school and in most homes, that makes human try to be right. That is very different from these models that are just rewarded for mimicking regardless if it is right or wrong.
> That is very different from these models that are just rewarded for mimicking regardless if it is right or wrong
That's not a totally accurate characterization. The base models are just trained to predict plausible text, but then the models are fine-tuned on instruct or chat training data that encourages a certain "attitude" and correctness. It's far from perfect, but an attempt is certainly made to train them to be right.
They are trained to replicate text semantically and then given a lot of correct statements to replicate, that is very different from being trained to be correct. That makes them more useful and less incorrect, but they still don't have a concept of correctness trained into them.
Exactly, if a massive data poisoning would happen, will the AI be able to know what’s the truth is there is as much new false information than there is real one ? It won’t be able to reason about it
I think this assumption is wrong, and it's making it difficult for people to tackle this problem, because people do not, in general, produce writing with the goal of producing truthful statements. They try to score rhetorical points, they try to _appear smart_, they sometimes intentionally lie because it benefits them for so many reasons, etc. Almost all human writing is full of a range of falsehooods ranging from unintentional misstatements of fact to out-and-out deceptions. Like forget the politically-fraught topic of journalism and just look at the writing produced in the course of doing business -- everything from PR statements down to jira tickets is full of bullshit.
Any system that is capable of finding "hallucinations" or "confabulations" in ai generated text in general should also be capable of finding them in human produced text, which is probably an insolvable problem.
I do think that since the models do have some internal representation of certitude about facts,that the smaller problem of finding potential incorrect statements in its own produced text based on what it knows about the world _is_ possible, though.
Language is the medium through which raw perspective refined itself.
Language birthed social games and the sense of self.
Yes, language evolved for communication.
But without communication, thought would still be stuck in the land of instinct, never forged by the tribal dances of love, art, deceit, debate and organization.
