The issue with term limits is that it gives a potential perception of bias related to whatever they do when they leave.
An example of this is in agencies like the FTC, where people leave the regulator and end up in well paid jobs in the companies they are supposed to be regulating. Even if there was no bias in their decision making people point to it and it undermines confidence in the system.
Can't you just make it so it's x years appointment, but you can't be reappointed?