Let's say the training corpus contains stories that compare example strategies. Each part of a strategy is explicitly weighed against another: one is called "superior".

Now all you need is a prompt that asks for "a strategy containing all superior features". There are probably plenty of grammatical examples elsewhere in the model that make that transformation.

All the work here is done by humans writing the training corpus. GPT never understood any of the steps. GPT just continued our story with the most obvious conclusion; and we made certain that conclusion would be correct.

GPT doesn't play games, it plays plays.

Perhaps the earlier or current variations of GPT, for most games? But the idea that LLMs can never make anything novel, that it will never "generalise out of distribution" (if that's the correct term here) seems to be just an assertion, not backed by any theory with great evidence behind it.

The "goal" of an LLM is to predict the next token. And the best way to do that is not brute force memorisation or regurgitating training data in various combinations, but to have a world model inside of it that will allow it to predict both the moves a bad player might make, and moves that a grandmaster might make.

That's another common misconception. That statement personifies GPT: GPT does not have goals or make predictions. Those are the effects of GPT: the behavior its authors hope will "emerge". None of that behavior comes from GPT itself. The behavior is defined by the patterns of tokens in the training corpus.

GPT itself has two behaviors: modeling and presentation. GPT creates an implicit model of every pattern it can find between the tokens in its training corpus. It then expands that model to include the tokens of an arbitrary prompt. Finally, it presents the model to us by starting at the location it just added the prompt tokens to, and simply following the most obvious path forward until that path ends.

The paths that GPT has available to present to us were already present in the training corpus. It isn't GPT that constructs the behavior, it is the people writing patterns into text.

> not brute force memorisation or regurgitating training data in various combinations

Not brute force: the combinations are not blindly assembled by GPT. GPT doesn't assemble combinations. The combinations were already assembled with patterns of grammar by the humans who wrote the valid progressions of game states. GPT found those patterns when it made its model.

> to have a world model inside of it that will allow it to predict both the moves a bad player might make, and moves that a grandmaster might make.

There is no prediction. A series of moves is a path carved into grammar. The path from one game state to the next involves several complex patterns that GPT has implicitly modeled. Depending on where GPT starts, the most obvious continuation may be to follow a more complex path. Even so, it's not GPT deciding where to go, it's the patterns that are already present that determine the path.

Because we use the same grammatical/writing patterns to describe "good play" and "bad play", it's difficult to distinguish between the two. GPT alone can't categorize the skill level of games, but narrative surrounding those game examples potentially can.