I don't consider you trespassing and the people who have the authority to remove you removing you as danger. Additionally, it's arguable he gave permission when they told him to leave or they'd remove him and he told them to remove him.
But their statement was just that they removed him from the talk without banning him from the conference, which turns out in practice to not be correct.
Not only did they have law enforcement intervene in his removal, but goons continued to threaten his removal and he was banned from the discord.
