If it does't have to be offline for long durations, software raid + adding a new drive every once in a while, and discarding failing drives is pretty foolproof.
AFAIK large data centers automate something like this.
The issue with (software) raid is you have no idea if what you're copying isn't actually corrupted. If the filesystem isn't checksumed there's no guarantee.
AFAIK large data centers automate something like this.