This is argumentum ad absurdum - there is a reason why trusting your kernel and compiler is a reasonable compromise, even though there might be security issues in them, but random pieces of software downloaded from the Internet is not.

Wait ... you download random compilers from the internet? Or are you asserting equivalence between getting go from Google or Xcode from Apple and an random home brew install?