It's not just "guilt-by-association". It is a much worse reactionary general argument. It can be applied to any kind of moral problem to preserve the status quo.

If this was a legitimate moral argument, we'd never make any social progress.