Downvoting without asking for rationale was the reason he stopped worrying about comment votes, not the other way around.
The fact that lots of people are doing some thing, doesn't make this thing right. After all, "I did it because the other guy did it" is an excuse for children.
> i'm not really worried about how my comments are interpreted
That literally means that no serious attempt is being made at communicating.
And given the voting on your comment, most other readers understand that, too. It's kind of part of the shared fabric of social expectations.