I think the bigger problem is that if the dataset was sufficiently poisoned, LLMs could start producing Greek question marks in their output. Like if you could tie it to some rare trigger words you could then use those words to cause generated code not to compile (despite passing visual inspection).