Have you seen the Othello paper? [1] To me it really puts paid to the idea that LLMs could just be stochastic parrots, in the sense of just rearranging things they've seen before. They at least can apply world models they devised during training (though whether or not one does for some given prompt can be a different question).

[1] https://thegradient.pub/othello/