Going back to the dinner analogy, you've stretched going to dinner with someone to happening to be in the same restaurant as that someone. It's not the same kind of association at all.
ex: "Nazis eat food, you eat food, therefore you're a Nazi."
Doing something that all people do and are expected to do, on either side of the conflict, does not actually make a difference. Being a complete and utter jackass makes the difference.