There a sort of necessary evil. They definitely complicate things, look at the debates over Tannenbaums work writing the os for secure enclave bios interactions. And, like protection rings they come with questions about bugs and subverted state.

I have worked on systems doing HSM like activity which depended on vendor specific TPM. They're a pain.