I suppose organizing departure in an adult manner (Offer some time to finish task, pass on knowledge, etc... AND let the people leave earlier if they prefer - you know, the whole "think about the team" thing) would also violate someone's meritocratic free speech ?
Do you really need to make it a freedom of speech issue? can't it be recognized as a human as a pile of shit to rug pull someone like that and not let them at least cancel their own talk?