Right. But if I were attacking you and was in your site and knew that you signed your hashes I'd be targeting your build script to save your key and passphrase, or build a backdoored version at the same time. Something like Certificate Transparency for code-signing would be really solid though; tampering would stand out. https://www.certificate-transparency.org/log-proofs-work

I think there's value in the purely user-based solution though. If two people go to a site at the same time and aren't served the same binary they should be curious why, even if the company securely and auditably signed both images.

And, it helps in all the cases when companies don't implement great release-security.