The point of a summarization model is if you have a thousand line change, it helps to have a one sentence explanation of what it is. The demo videos the author used here really don't do a good job communicating that, because the summary GPT-3 wrote for his one line commit was longer than the commit itself.
Right, and even if GPT-3 could summarize the thousand-line diff in a sensible way, without introducing any falsehoods, it would still be strictly worse than the developer writing a sentence explaining what they think they've accomplished with the commit.
It's just the same thing as with comments and "self-documenting code". The code tells you what (and if written carefully, it may be even somewhat effective at it). It can't tell you why. Neither can a GPT-3 summary of it.
