I don't think that is correct.

LLMs are literally predicting the outcome of an experiment really well constantly, yet they are best described as pretending to understand really, really well ...

People who fake understanding by creating an ad hoc internal model to sound like an expert often get a lot of things right. You catch such people by asking more complex questions, and then you often get extremely alien responses that no sane person would say if they understood. And yes, LLMs give such responses all the time, every single one of them, none of them really understands much, they have memorized a ton of things and have some ad hoc flows to get through simple problems, but they don't seem to really understand much at all.

Humans who just try to sound like an expert also make similar alien mistakes as LLMs do, so I think since we say such humans don't learn to understand we can also say that such models don't learn to understand. You don't become an expert by trying to sound like an expert. These models are trained to sound like experts, so we should expect them to be more like such humans rather than the humans who become experts.

Hmmm, I don't agree. They do seem to understand some things.

I asked ChatGPT the other day how fast an object would be traveling if "dropped" from the Earth with no orbital velocity, by the time it reached the sun. It brought out the appropriate equations and discussed how to apply them.

(I didn't actually double-check the answer, but the math looked right to me.)

It also seems to have a calculation or "analysis" function now, which gets activated when asking it specific mathematical questions like this. I've imagined it works by using the LLM to set up a formula, which is then evaluated in a classical way.

There are limits on what it can do, just like any human has similar limits. ChatGPT can answer more questions like this correctly than the average person could from off the street. That seems like understanding to me.

LLMs are the reason everyone is suddenly taking seriously the ideas that machines can be intelligent. Government officials, non-tech pundits, C-suite inhabitants, average people, everyone is now concerned about AI and its implications.

And I would say that's because of LLMs ability to predict the outcome of an experiment really well.

As for the "pretending", I think that comes from the fact LLMs are doing something quite different than humans to produce language. But that doesn't make them unintelligent. Just makes them not human.

> Government officials, non-tech pundits, C-suite inhabitants, average people, everyone is now concerned about AI and its implications.

All you need for that is for the AI to talk like experts, not for the AI to be experts. AI talking like experts without understanding much maps very well to what we see today.

Does Newton's Theory of Gravity actually understand gravity or is it just a really good tool for 'predicting outcomes'. The understanding is baked in to the formula and is a reflection of what Newton experienced in reality.

Newton's theory is ultimately and slightly wrong but still super useful and many LLMs are basically like this. I can see why all this becomes confusing but I think part of that is when we anthropomorphic words to describe these things that are just math models.

Most of the "predictions" you'd make using it, when discovered, were wrong. it is a deeply unpredictive formula, being useless for predicting vast classes of problems (since we are largely ignorant about the masses involved, and cannot compute the dynamics beyond a few anyway).

Science is explanatory, not "predictive" -- this is an antique mistake.

As for 'math models' insofar as these are science, they arent math. They use mathematical notation as a paraphrase for english, and the english words refer to the world.

F=GMM/r^2 is just a summary of "a force occurs in proportion to the product to two masses and inversely in proportion to their square distance"

note: force, mass, distance, etc. <- terms which describe reality and its properties; not mathematics.

Regarding your last statement, what do you see as the distinction?

Take for example eg continuity. Students are first taught it means you can graph a function in a single stroke of the pen. Later comes epsilon and delta, this is more abstract but at that stage the understanding is that "nearby values map to nearby values" (or some equivalent).

If the student dives in from there, they're taught the "existence" of real numbers (or any mathematical term) is rather a consequence of a system of symbols and relations that increasingly look nothing like "numbers", instead describing more of a process.

Later that "consequence" and "relation" themselves are formalities. "Pure" math occasionally delivers strange consequences in this sense. But it always boils down to a process that something or another must interpret and carry out.

So I wonder whether the edifice is meaningfully a thing in and of itself. Methods developed in ancient China and India etc would have been useful to the Greeks and vice versa, however all of them though worked by means of the human brain. "Line" has a distinct meaning to us, the axioms of geometry don't create the line, they allow us to calculate some properties more efficiently. We always need to interpret the result in terms we understand, don't we?

>Science is explanatory, not "predictive" -- this is an antique mistake.

I see no distinction. If something can explain it can 'predict'.