> It does a good job of summarizing API docs and stackoverflow posts, and even gives me snippets of code. I had it generate Python scripts to do simple tasks.

That's because these are highly uniform, formulaic and are highly constrained both linguistically and conceptually.

It's basically doing (incredibly sophisticated) copying and pasting.

Try asking it to multiply two random five digit numbers. When it gets it wrong, ask it to explain how it did it. Then tell it it's wrong, and watch it generate another explanation, probably with the same erroneous answer. It will keep generating erroneous explanations for the wrong answer.