Sounds suspicious to me.

At a quick glance, most benchmarks show only 50% to 60% single threaded improvement between M1 and M4.

That wouldn't explain compilation times going from 6 minutes to 2 minutes.