I think it's generally because the balance point isn't a stable equillibrium. There's no force that encourages balance - it would have to be serendipitous. That's unlike the extremes, where a thriving interstellar civilization likely leads to mass colonization or that life is hard and typically goes extinct before then.
The time scales in these sorts of estimates are enormous, often tens or hundreds of times the span we needed to rise up. So if we even have an inkling to be expansionary (and we do), and aliens are like us in wanting to thrive comfortably (with enough space per unit being), then it's likely they too would have at least a mild drive to expand. Over mnay millions of years, even a very mild push out would fill a galaxy.
I don't get it why so many people either want life to be basically on every second star or not exist outside Earth at all.
Let's just accept that Fermi and all others hugely overestimated the chances and roll with it, will be far more productive imo.
So with the GRBs in mind, we can safely narrow livable space to outskirts of Milky Way and focus our search there.