Actually, it is not hate speech to disagree with the Jewish religion or specific customs (that is protected instead by freedom of religion, which is a different argument, and an important one, but probably not with the same weight as basic personhood). That's not what people usually talk about when they mention antisemitism. Antisemitism, taken in the Nazi way, is the 'disagreement' over whether Jewish people as an ethnic group deserve rights as people or citizens.

Edit: But even with that said, I agree restrictions to speech should rather be too few than to many, and too narrow rather than too broad. Which is why I think the line should never remotely try to cover every degree of racism or xenophobia at all. I could be convinced of the need to restrict public calls for genocide, for example, though.