It is slow, and yes it is a problem, but given that typical Rust code generally needs fewer full compiles to get working tests (with more time spent active in the editor, with an incremental compiler like Rust Analyzer) it usually balances out.

Cargo also has good caching out of the box. While cargo is not the best build system, it's an easy to use good system, so you generally get good compile times for development when you edit just one file. This is along made heavy use of with docker workflows like cargo-chef.