I find it distracting, I think because I'm so used to humans using two hyphens as a substitute, sometimes with spaces around them, that a real em-dash used what I suppose is correctly looks too small and tight.

Yeah, understandable.

I’ve actually never been a fan of the “correct usage” that forgoes any spacing around it. I find it crowded and jarring.

I like to use a “thin space” around an em dash — it breaks it up without adding too much space.