I gave it a different kind of puzzle, again with a twist (no solution), and it spit out nonsense. "I have two jars, one that can hold 5 liters, and one that can hold 10 liters. How can I measure 3 liters?" It gave 5 steps, some of which made sense but of course didn't solve the problem. But at the end it cheerily said "Now you have successfully measured 3 liters of water using the two jars!"
That's a good example which illustrates that GPT (regardless of the number) doesn't even try to solve problems and provide answers, because it's not optimized to solve problems and provide answers - it is optimized to generate plausible text of the type that might plausibly be put on the internet. In this "genre of literature", pretty much every puzzle does have a solution, perhaps a surprising one - even those which are logically impossible tend to have actual solutions based on some out-of-box thinking or a paradox; so it generates the closest thing it can, with a deus ex machina solution of magically getting the right answer, since probably even that is more likely as an internet forum answer as proving that it can't be done. It mimics people writing stuff on the internet, so being wrong or making logic errors or confidently writing bullshit or intentionally writing lies all is plausible and more common than simply admitting that you have no idea - because when people have no idea, they simply don't write a post about that on some blog (so those situations don't appear in GPT training), but when people think they know, they write it up in detail in a confident, persuasive tone even if they're completely wrong - and that does get taught to GPT as an example of good, desirable output.
