That's not at all analogous to what I'm talking about. The comparison would be picking an LLM or a random person out of the phone book to, say, operate a vending machine... and we already know LLMs are unable to do that, given the results of Vending-Bench.
More than 10% of the global population is illiterate. Even in first world countries, numeracy rates are 75-80%. I think you overestimate how many people could pass the benchmark.
Edit - rereading, my comment sounds far too combative. I mean it only as an observation that AI is catching up quickly vs what we manage to teach humans generally. Soon, if not already, LLMs will be “better educated” than the average global citizen.
And yet, I would be completely confident that an average illiterate person could pass the Vending-Bench test indefinitely if you gave them interfaces that don't depend on the written word (phone calls, abacuses, piles of blocks, whatever), and that the "smartest" LLM in the world couldn't. It's not about level of education, beyond the bare minimum needed to have any kind of mental model of the world.
