You're right that in a universe where everyone grants their end users necessary freedoms and don't use coercion to limit those rights, GPL would be completely unnecessary. Until that happens, coercing people to stop further coercion is the best thing we have.
I'm taking about a universe where violence is impossible. Not a universe where people are compelled to distribute source code whenever they have distributed a binary file.
I'm fundamentally opposed to the idea of forcing people to do what I want and that's the clear purpose of GPL.
Somebody not doing what I want is not coercing me. I'm free to use or not use a binary distributed without it's source code, or I can try to reverse engineer it. Forcing them to do the additional work of sending me the source code would be coercing them, it would be impossible in my hypothetical alternative universe, and it would be wrong in this universe.