Are you one of those people? how can you be so confident? I think everyone should have updated their priors after how surprising the emergent behavior in GPT3+ are
I don't think GPT3's "emergent behavior" was very surprising, it was a natural progression from GPT2, and the entire purpose of GPT3 was to test the assumptions about how much more performance you could gain by growing the size of the model. That isn't to say GPT3 isn't impressive, but its behavior was within the cone of anticipated possibilities.
Based on a similar understanding, the idea that transformer models will lead to AGI seems obviously incorrect, as impressive as they are, they are just statistical pattern matchers of tokens, not systems that understand the world from first principles. And just in case you're among those that believe "humans are just pattern matchers", that might be true, but humans are modeling the world based on real time integrated sensory input, not on statistical patterns of a selection of text posted online. There's simply no reason to believe that AGI can come out of that.
Please read the paper. The authors are using more precise and specific metrics that qualitatively measure the same thing. Instead of having exact string match being 1 if 100% correct, 0 if there is any failure, they use per-token error. The crux of their argument is that per-token error is a better choice of metric anyway, and the fact that "emergent abilities" do not occur when using this metric is a strong argument that those abilities don't really exist.
However thermal energy does not more precisely or specifically measure a phase transition. They are only indirectly linked - nobody would say that thermal energy is a better measure of state-of-matter than solid/liquid/gas. Your argument makes absolutely zero sense. Frankly it seems intentionally ignorant.
Per token error is a fairly useless metric. It's not predictive and it tells you absolutely nothing.
They say it's a superior metric but clearly the wider research community disagrees since no one has cared to adopt per token error as a metric in subsequent papers.
>and the fact that "emergent abilities" do not occur when using this metric is a strong argument that those abilities don't really exist.
If your conclusion is that those abilities don't exist then you clearly didn't read the paper very well.
They never argue those abilities don't exist, they simply argue whether we should call them "emergent" or not.
>However thermal energy does not more precisely or specifically measure a phase transition. They are only indirectly linked - nobody would say that thermal energy is a better measure of state-of-matter than solid/liquid/gas. Your argument makes absolutely zero sense. Frankly it seems intentionally ignorant.
Phase Changes are literally driven by changes in thermal energy.
Water boils when it absorbs enough thermal energy to break intermolecular forces keeping its liquid state together.
solid/liquid/gas is descriptive. It's not a measure of anything.
Anyway, the point is simple. Despite thermal energy driving state change after a certain threshold, that "point" doesn't look like anything special.
Smooth quantitative change sometimes results in sudden qualitative changes.
