Have you profiled the build? Is most of it spent in the compiler?
A more practical point is that the main competitor is C++, a language notorious for long compilation times, so not-ultra-fast compile times might not be the highest priority for people working on rustc.
C++ compilers are very fast (I would say amazingly fast) unfortunately atrocious header only implementations like Boost drags it down. I just did clang++ -E on a single #include of cpp-netlib which depends on Boost asio and the dumped output came to:
$ wc test1
279506 897765 10174700 test1
And this is after enabling dynamic linking which cuts down a few thousand lines. Library writers are not giving compiler writers a break! I find this practice atrocious. People should be able to just import the interface instead the whole implementation of everything (and enable "headers only" feature at the end if so desired to eliminate the need of linking).
Compatibility with C tooling is also a big reason, as templates can only be header only to be consumed by other translation units.
Unfortunately "export template" was a failed experiment.
Now C++ developers need to wait until C++17 for modules, if they ever get into the standard. And if they do, most likely it will take until around 2020 for all major C++ compilers across embedded, desktop and server systems offer support for it.
Now the question is, if one is willing to wait that long or rather use a language that can use modules today.
Err, header-only libraries exist mostly because templates require them, not because people don't like linking (well, there's some of that too, but it's the minority).
While the problems with templates and huge includes are undeniable, I've found in my experience that for a decently size C++ program the linking stage alone can take way more than the 3-4 seconds the GP is talking about.
C and C++ are definitely some hard beasts to compile, even without boost.
A more practical point is that the main competitor is C++, a language notorious for long compilation times, so not-ultra-fast compile times might not be the highest priority for people working on rustc.