That's a bit unfair to the modern compilers - there's a lot more standards to adhere to, more (micro)architectures, frontends need to plug into IRs into optimisers into codegen, etc. Some of it is self-inflicted: do you need yet-another 0.01% optimisation? At the cost of maintainability, or even correctness? (Hello, UB.) But most of it is just computers evolving.
Indeed, and thankfully there exist languages like D and Delphi to prove as being a modern compiler and fast compilation times are still possible 40 years later.
But those are not rules. If you're doing stuff for fun, check out QBE <https://c9x.me/compile/> or Plan 9 C <https://plan9.io/sys/doc/comp.html> (which Go was derived from!)