> it is completely impossible to gain knowledge you do not already possess from a language model

Not true. Emergent abilities is an active research area in LLMs [0]. They even have pretty graphs on the topic.

[0] https://ai.googleblog.com/2022/11/characterizing-emergent-ph...