Yeah, but with binary packages you can add another lay of defense in depth, signed packages, signature checking, etc. It's not just about the original authors themselves, it can also be about attacks on the public repositories, for example.