It's self-evident that GPT is a world-modeller, at least within the confines of the text boundary. It's able to come up with novel ideas seen nowhere in the training data, combinations that demonstrate there is a world concept web and not just a text probability web. It may not "understand" much of the hallucination nonsense it spits out, but there absolutely are moments where it "understands".

See the Rome example on this page: https://oneusefulthing.substack.com/p/feats-to-astonish-and-... This is essentially a completely novel answer to an /r/AskHistorians style question, which I would consider one of the most difficult types of internet text to model, in terms of the amount of understanding and concept webs you need to tie together

Here's another example of GPT-4 doing non-trivial world modelling: How would three philosophers review the TV show Severence? https://i.imgur.com/FBi31Qw.png

The Othello-GPT experiment (https://thegradient.pub/othello/) probably still is the most relevant argument about these models' capabilities of building an internal world model.

Doesn't that "It is modelling the various text generation processes that lead to the contents of the internet," seems tautologically true?

(I'm not the person who wrote the grandparent of the present comment.)