Backups sent to a remote system should be append only so that if a machine is compromised, a malicious actor cannot delete or corrupt previous backups.
An easy way is to stream the backups to tape, and have someone swap the tapes. Typically people who do this take the recorded tapes offsite, but to prevent against an electronic attack it’s adequate to have the removed tapes even in the same room, just not in the tape drive.
Separate storage of file hashes would make tampering detectable. Actually preventing root modifying a file probably has to be done at the hardware level.
How do you accomplish this? I have some ideas/assumptions, but I'd like to hear more about the best practices here.