It's not about struggling so much as about avoiding a situation where one small problem makes a site either look visually 'broken' or (worse) fail to function correctly.
There are way too many small bugs and incompatibilities like these, and way, way too many browser versions still seeing use by at least a small percentage of users, to worry about them individually, unless you want that to eat up all your time.
So your choice is between "be okay with a broken experience for some users", or "use build tools to compile to a common-ancestor target of near-universally-supported CSS and JS".
What build tools do you use that ensure your site works consistently across different browser versions without needing any browser-version-specific testing/fixes on your part?
I've worked on web pages that use what seems to be a pretty standard Webpack/Babel/PostCSS/etc setup, and they still ran into some issues where layout or behavior was different between browsers. Either because they used a CSS property value or JavaScript API (or API parameter/option) that wasn't available everywhere, or because of more subtle disagreements between browsers about things like where an inline element's containing block is.
You're right, it's not possible to eliminate the need for testing/fixes. I didn't say it was. It's a tradeoff; compiling to a stable older set of features reduces bugs but doesn't eliminate them.
Part of the work has to be on the developer. There are certain properties that will screw up positioning when used with flex elements in IE, for example, and knowing what those are and how to avoid those combinations is part of the cost of skillfully using flexbox. The compiler can't always do the work, but we should let it do as much as it can.
There are way too many small bugs and incompatibilities like these, and way, way too many browser versions still seeing use by at least a small percentage of users, to worry about them individually, unless you want that to eat up all your time.
So your choice is between "be okay with a broken experience for some users", or "use build tools to compile to a common-ancestor target of near-universally-supported CSS and JS".