In my experience it generally has a very good understanding and does generate the relevant test cases. Then again I don't give it a grammar, I just let it generalize from examples. In my defense I've tried out some very unconventional languages.
Grammars are an attempt at describing a language. A broken attempt if you ask me. Humans also don't like them.
For natural language you are right. The language came first, the grammar was retrofitted to try to find structure.
For formal languages, which programming languages (and related ones like query languages, markup languages, etc) are an instance of, the grammar defines the language. It come first, examples second.
Historically, computers were very good at formal languages. With LLMs we are entering a new age where machines are becoming terrible at something they once excelled at.
Have you lately tried asking Google whether it's 2025? The very first data keeping machines (clocks) were also pretty unreliable at that. Full circle I guess.
Grammars are an attempt at describing a language. A broken attempt if you ask me. Humans also don't like them.