I'd agree with you in the past, when KiCAD didn't work as nicely with git. And for a professional project, I'd still at least keep snapshots of every milestone with "compiled" output.

But for a hobby project I really don't see the need. It takes a few extra minutes to install KiCAD, clone the repo, and view the schematic at any point in time.