“ E.g. punishment for the sake of retribution is near impossible to morally justify if you don't believe in free will because it means you're punishing someone for something the had no agency over.”

False, the punisher also has no will, so it doesn’t matter.