https://amontalenti.com/2024/06/19/backups-restic-rclone

The tools I use on Linux for backup are restic + rclone, storing my restic repo on a speedy USB3 SSD. For offsite, I use rclone to incrementally upload the entire restic repository to Backblaze B2.

The net effect: I have something akin to Time Machine (macOS) or Arq (macOS + Windows), but on my Linux laptop, without needing to use ZFS or btrfs everywhere.

Using restic + some shell scripting, I get full support for de-duplicated, encrypted, snapshot-based backups across all my "simpler" source filesystems. Namely: across ext4, exFAT, and (occasionally) FAT32, which is where my data is usually stored. And pushing the whole restic repo offsite to cloud storage via rclone + Backblaze completes the "3-2-1" setup straightforwardly.

Then again, the likelihood that the backup will be inconsistent is fairly low for a desktop, so it’s probably fine.

I think the optimal solution is:

1) file system level atomic snapshot (ZFS, BTRFS etc)

2) Backup the snapshot at a file level (restic, borg etc)

This way you get atomicity as well as a file-based backup which is redundant against filesystem-level corruption.

In my linked post above, I wrote about this:

"You might think btrfs and zfs snapshots would let you create a snapshot of your filesystem and then backup that rather than your current live filesystem state. That’s a good idea, but it’s still an open issue on restic for something like this to be built-in (link). There’s a proposal about how you could script it with ZFS in this nice article (link) on the snapshotting problem for backups."

The post contains the links with further information.

My imperfect personal workaround is to run the restic backup script from a virtual console (TTY) occasionally with my display server / login manager service stopped.

I've been thinking of somehow triggering restic by zrepl whenever it takes a snapshot, but I haven't figured a way of securely grabbing credentials for it to unlock the repository and to upload to s3 without requiring user intervention.

This also helps dealing with run-away (or long running) processes eating disk space, as you always have some extra space set aside..

(Also I think lvm doesn't need the volume blocks to be contiguous on the physical volume. So you might have N free space after volume a and M after volume b, and lvm would let you create a new N+M sized volume.)

As I said: garbage for snapshots

While Linux is great at many things, backups is one area I find lacking compared to what I'm used to from Windows. There I take frequent incremental whole-disk backups. The backup program uses the Volume Shadow Copy Service to provide a consistent state (as much as possible). Being incremental they don't take much space.

If my disk crashes I can be back up and running like (almost) nothing happened in less than an hour. Just swap out the disk and restore. I know, as I've had to do that twice.

[1]: https://learn.microsoft.com/en-us/windows/win32/vss/the-vss-...

[2]: https://learn.microsoft.com/en-us/windows/win32/vss/overview...

Ie automatically creates a snapshot and sends the incremental changes since previous snapshot to a backup destination like a NAS or S3 blob storage.

https://github.com/tasket/wyng-backup

edit: requires lvm thin provisioned volumes

There is also thin-send-recv which basically does the same as zfs send/recv just with lvm:

https://github.com/LINBIT/thin-send-recv

it uses the same functions of the device mapper to allow incremental sync of lvm thin volumes.

It's just such a low-effort peace of mind. Just a few clicks and I know that regardless what happens to my disk or my system, I can be up and running in very little time with very little effort.

On Linux it's always a bit more work, but backups and restore is one of those things I prefer is not too complicated, as stress level is usually high enough when you need to do restore to worry about forgetting some incantation steps.

Backing up a linux system in combination with REAR:

https://github.com/rear/rear

and a backup utility of your choice for the regular backup has never failed me so far. I used it to restore linux systems to complete different hardware without any troubles.

While I've had to recover in anger twice, I've used the same procedure to migrate to new hardware many times. Just restore to the new disk in the new machine, and let Windows reboot a few times and off I went.

REAR looks useful, hadn't seen that before.

I just make a full dedupped backups from LVM snapshots with kopia, but I've set that up only on one system, on others I just use kopia as-is.

It takes some time, but that's fine for me. Previous backup of 25 GB an hour ago took 20 minutes. I suppose if it only walked files it knew were changed it would be a lot faster.

> I suppose if it only walked files it knew were changed it would be a lot faster.

Right, for me I'd want to set it up to do the full disk, so could be millions of files and hundreds of GB. But this trick should work with other backups software, so perhaps it's a viable option.

Here's the script, should it be of benefit to someone, even if it of course needs to be modified:

    #!/bin/sh
    success=false
    teardown() {
      umount /mnt/backup/var/lib/docker || true
      umount /mnt/backup/root/.cache || true
      umount /mnt/backup/ || true
      for lv in root docker-data; do
        lvremove --yes /dev/hass-vg/$lv-snapshot || true
      done
    
      if [ "$1" != "no-exit" ]; then
        $success
        exit $?
      fi
    }
    
    set -x
    set -e
    teardown no-exit
    trap teardown EXIT
    for lv in root docker-data; do
      lvcreate --snapshot -L 1G -n $lv-snapshot /dev/hass-vg/$lv
    done
    
    mount /dev/hass-vg/root-snapshot /mnt/backup
    mount /dev/hass-vg/docker-data-snapshot /mnt/backup/var/lib/docker
    mount /root/.cache /mnt/backup/root/.cache -o bind
    
    chroot /mnt/backup kopia --config-file="/root/.config/kopia/repository.config" --log-dir="/root/.cache/kopia" snap create / /var/lib/docker
    kopia --config-file="/root/.config/kopia/repository.config" --log-dir="/root/.cache/kopia" snap create /boot /boot/efi
    success=true

e.g. The database might append to write-ahead logs in a different order than the order in which the backup tool reads them.

The main difference is that Windows and MacOS have a mechanism that communicates with applications that a snapshot is about to be taken, allowing the applications (such as databases) to build a more "consistent" version of their files.

In theory, of course, database files should always be in a logically consistent state (what if power goes out?).

Well, supported by Windows and MacOS. Linux only if you happen to use zfs or btrfs, and also only if the backup tool you use happens to rely on those snapshots.

The GP is talking about when the backup software reads database files gradually from the live filesystem at the same time as the database is writing the same files. This can result in an inconsistent "sliced" state in the backup, which is different from anything you get if the database crashes or the system crashes or loses power.

The effect is a bit like when "fsync" and write barriers are not used before a server crash, and an inconsistent mix of things end up in the file. Even databases that claim to be append-only and resistant to this form of corruption usually have time windows where they cannot maintain that guarantee, e.g. when recycling old log space if the backup process is too slow.

https://github.com/creativeprojects/resticprofile

I also use Kopia as an alternative to Restic, in case some critical bugs happen to either one of them.

https://kopia.io/

I found their explanation here:

https://github.com/kopia/kopia/issues/1764

https://github.com/kopia/kopia/issues/544

Still not solved after many years :(

Now I use Borg + Restic and I am happy

+ GUI for Restic https://github.com/garethgeorge/backrest

+ GUI for Borg https://github.com/borgbase/vorta

- encrypted, chunk-deduped, snapshotted backups

- single Go binary, so I could even backup the binary used to create my backups

- reasonable versioning and release scheme

- I could read, and understand, its design document: https://github.com/restic/restic/blob/master/doc/design.rst

I then just tried using it for a year and never hit any issues with it, so kept going, and now it's 6+ years later.

1 - some of my first experiences with backup failures were due to media problems -- this was back in the days when "backup" pretty much meant "pipe tar to tape" and while the backup format was simple, tape quality was pretty bad. These days, media -- tape or disk -- is much more reliable, but backup formats are much more complex, with encryption, data de-dup, etc. Therefore, I consider the backup format to be at least as much of a risk to me now as the media. So, anyway, I do two backups: the local one uses restic, the cloud backup uses borg.

2 - I use rsync.net, which I generally like a lot. I wrote up my experiences with append-only backups, including what I did to make them work with rsync.net here: https://marcusb.org/posts/ransomware-resistant-backups/

"My reasoning for splitting these two processes — restic backup and rclone sync — is that I run the local restic backup procedure more frequently than my offsite rclone sync cloud upload. So I’m OK with them being separate processes, and, what’s more, rclone offers a different set of handy options for either optimizing (or intentionally throttling) the cloud-based uploads to Backblaze B2."

Borgbackup + rclone (or aws) [1]

It works so well, I even use this same script on my work laptop(s). rclone enables me to use whatever quirky file sharing solution the current workplace has.

[1]: https://github.com/kmARC/dotfiles/blob/master/bin/backup.sh

[1] https://github.com/kimono-koans/httm

On Linux, the system I liked best was rsnapshot: I love its brutal simplicity (cron + rsync + hardlinks), and how easy it is to browse previous snapshots (each snapshot is a real folder with real files, so you can e.g. ripgrep through a date range). But when my backups grew larger I eventually moved to Borg to get better deduplication + encryption.

Let's say you're running Fedora with Gnome and you want to switch to KDE without doing a fresh install. You make a backup, then go through the dozens of commands to switch, with new packages installed, some removed, display managers changed etc. Now something doesn't work. Would recovering from the restic backup reliably bring the system back in order?

The tool from the original post seems to be geared towards that, while most Restic and rclone examples seem to be geared towards /home backup, so I wonder how much this is actually an alternative.

I think of restic system backups covering something like losing a customized /etc file in an apt upgrade and wanting to get it back.

Snapper on openSUSE is integrated with both zypper (package manager) and YaST (system configuration tool) [1], so you get automatic snapshots before and after destructive actions. Also, openSUSE defaults to btrfs, so the snapshots are filesystem-native.

[0]: http://snapper.io/

[1]: https://en.opensuse.org/Portal:Snapper

It is when the defaults make the parts integrate nicely like this that the “greater is more than the sum of its parts” come into place.

Has anyone figured out an easy way to get this back with systemd-boot?

https://news.opensuse.org/2024/03/05/systemd-boot-integratio...

https://en.opensuse.org/Systemd-boot#Installation_with_full_...

https://github.com/openSUSE/sdbootutil

I haven't tried it though so I don't know for sure. (I have my own custom systemd-boot setup that predates theirs, and since my setup uses signed UKIs and theirs doesn't, I don't care to switch to theirs. I can still switch snapshots manually with `btrfs subvol` anyway; it just might require a live CD in case the default snapshot doesn't boot.)

Currently this needs to be set up semi-manually (select some options in the installer, then run some commands after install), but it'll be automatic soon.

Because it's not very popular in the US which has mostly cemented around fedora/ubuntu/arch so you don't hear much about any other distros, and most other countries around the world tend to just adopt what they learn from the US, due to the massively influential gravitational field the US has on the tech field.

But in the german speaking world many know about it. It's a shame that despite the internet being relatively borderless it's still quite insular and divided. I'm not a native german speaker but it helps to know it since there's a lot of good linux content out there that's written in german.

[0]: https://gitlab.com/btrfs-assistant/btrfs-assistant

https://github.com/snapshotmanager/boom-boot

I have used Linux for 10+ years but over the I have spent hours, days and weeks trying to undo or fix little issues I introduce by tinkering around with things. Often I seem to break things at the worst times, right as I am starting to work on some new project or something that is time sensitive.

Now, I can just roll back to an earlier stable version if I don't want to spend the time right then on troubleshooting.

I've enabled this on all my family members machines and teach them to just roll back when Linux goes funky.

It’s not that I couldn’t retype the config file I accidentally wrote over while tinkering, but I like the safety that comes with Timeshift to try and fail a few times.

Hard lessons come hard. This softens those lessons a little while maintaining the learning.

Linux is simultaneously the most configurable and the most brittle OS IMHO. NixOS takes away all the brittleness and leaves all the configurability, with the caveat that you have to declaratively configure it using the Nix DSL.

Similarly, it doesn't do anything in regards to user files.

https://nix-community.github.io/home-manager/index.xhtml#sec...

To me, it's better than a filesystem-backup because the things that make it into home manager tend to be exactly the things that I want to back up. The rest of it (e.g. screenshots, downloads) aren't something I'd want in a backup scheme anyhow.

Code and docs are in source control. My phone syncs images to PCloud when I take them. Anything I download is backed up... wherever I downloaded it from.

I'm honestly curious what packages you have a problem with

Skype, Steam, and Lightworks are all directly available in the repos and seem to work fine as far as I can tell. I'm sure there are proprietary packages that don't work or aren't in the repo, but I haven't really encountered them.

For the system itself, each time you run "nixos-rebuild switch" it builds a system out of your configuration.nix, including an activation script which sets environment variables and symlinks and stops and starts services and so on, adds this new system to the grub menu, and runs the activation script. It specifically doesn't delete any of your old stuff from the nix store or grub menu, including all your older versions of packages, and your old activation scripts. So if your new system is borked you can just boot into a previous one.

NixOS does that. And I'm pretty sure that no other flavor of Linux does. First time I realized I could just blithely "shop around window managers" simply by changing a couple of configuration lines, I was absolutely floored.

NixOS is the first Linux distro that made me actually feel like I was free to enjoy and tinker with ALL of Linux at virtually no risk.

There is nothing else like it. (Except Guix. But I digress.)

Obviously this is just opinion (no need for someone to supply nuance) but from my perspective the NixOS model is so obviously the "correct" way of doing an OS that it really annoys me that it's not the standard for every operating system. Nix itself is an annoying configuration language, and there are some more arcane parts of config that could be smoothed over, but the model is so obviously great that I'm willing to put up with it. If nothing else, being able to trivially "temporarily" install a program with nix-shell is a game-changer to me; it changes the entire way of how I think about how to use a computer and I love it.

Flakes mostly solve my biggest complaint with NixOS, which was that it was kind of hard to add programs that weren't merged directly into the core nixpkgs repo.

- Literally every person who's read the Nix paper and drank the kool-aid thinks this lol.

I STILL don't completely understand every element of my nix config but it's still quite usable. Adding software requires adding it to the large-ish config file, largely because I created overlay namespaces of "master.programname", "unstable.programname" and "stable.programname" (with the default being "unstable" in my case) but those would all ideally be moved out into 2 text files, 1 for system level (maybe called system_packages.txt) and one for a named user (perhaps called <username>_packages.txt) and if those could be imported somehow into the configuration.nix, I think that would make things a bit easier for end-users, at least initially.

The commandline UI (even the newer `nix` one) could still use an overhaul IMHO. The original CL utils were CLEARLY aimed directly at Nix developers, and not so much at end-users...

I've been working on my own wrapper to encapsulate the most common use-cases I need the underlying TUI for https://github.com/pmarreck/ixnay < and that's it so far.

This makes no sense! System Restore is a useless wart that just wastes time making "restore points" at every app/driver install and can rarely (if ever) produce a working system when used to "restore" anything. It does not back up user data at all. Time Machine is a whole-system backup solution that seems to work quite well and does back up user data.

To me the quoted statement might as well read "a tool similar to knitting needles (in hobby shops) and dremels (in machine shops)"

Reading their description further, it seems like they are implementing something similar to TimeMachine (within the confines of what linux makes possible), and not at all like "System Restore". This seems sane as this implements something that is actually useful. They, sadly, seem to gloss over what the consequences are of using non-btrfs FS with this tool, only mentioning that btrfs is needed for byte-exact snapshots. They do not mention what sort of byte-inexactness ext4 users should expect...

Which files are part of a shadow copy is determined by the one creating a shadow copy, so it could include user data.

You can view and access the files in a shadow copy using ShadowExplorer[3] if you don't have the pro versions.

[1]: https://learn.microsoft.com/en-us/windows-server/storage/fil...

[2]: https://learn.microsoft.com/en-us/windows/win32/vss/the-vss-...

[3]: https://www.shadowexplorer.com/

Newer versions of Windows have File History to backup user data (I don't think they have an integrated system/file solution quite like Time Machine though).

However it makes some sense to keep system/user data separate. You don't want to lose your doc edits because you happened to have a bad driver upgrade at the same time. Likewise, you don't want to roll your entire system back to get an old version of a doc.

Time Machine is trivial to implement (without the UI) with disk snapshots (that's what it does--store disk snapshots to an external disk)

The license conflict and OpenZFS always having to chase kernel releases often resulting in delayed releases for new kernels means I cannot confidently use them with rolling release distros on the boot drive. If I muck something up, the data drives will be offline for a few minutes till I fix the problem. Doing the same with the boot drive is pain I can live without.

A shame most distro's installers don't support it natively, but an encrypted rootfs on ZFS is great once you get it setup.

I am somewhat wary of trying this, mucking something up and wasting a lot of time wrestling with it. Will probably play around with it in a vm and use it during the next ssd upgrade.

Would have been so much better if the distros showed more interest in ZFS

That said, I've been using the tool for a while now and it's been really rock solid. And once you have it installed and working, you don't really have to touch it again, until some hypothetical time when a new backward-incompatible zpool feature gets added that you want to use, and you need a newer ZFSBootMenu build to support it.

Because it's just an upstream Linux kernel with the OpenZFS kmod, and a small dracut module to import the pool and display a TUI menu, it's mechanically very simple, and relying on core ZFS support in the Linux kernel module and userspace that's already pretty battle tested.

After seeing people in IRC try to diagnose recent GRUB issues with very vanilla setups (like ext4 on LVM), I'm becoming more and more convinced that the general approach used by ZFSBootMenu is the way to go for modern EFI booting. Why maintain a completely separate implementation of all the filesystems, volume managers, disk encryption technologies, when a high quality reference implementation already exists in the kernel? The kernel knows how to boot itself, unlock and mount pretty much any combination of filesystem and volume manager, and then kexec the kernel/initrd inside.

The upsides to ZFSBootMenu, OTOH,

    * Supports all ZFS features from the most recent OpenZFS versions, since it uses the OpenZFS kmod 
    * Select boot environment (and change the default boot environment) right from the boot loader menu
    * Select specific kernels within each boot environment (and change the default kernel)  
    * Edit kernel command line temporarily  
    * Roll back boot environments to a previous snapshot  
    * Rewind to a pool checkpoint  
    * Create, destroy, promote and orphan boot environments  
    * Diff boot environments to some previous snapshot to see all file changes
    * View pool health / status  
    * Jump into a chroot of a boot environment  
    * Get a recovery shell with a full suite of tools available including zfs and zpool, in addition to many helper scripts for managing your pool/datasets and getting things back into a working state before either relaunching the boot menu, or just directly booting into the selected dataset/kernel/initrd pair.
    * Even supports user mode SecureBoot signing -- you just need to pass the embedded dracut config the right parameters to produce a unified image, and sign it with your key of choice. No need to mess around with shim and separate kernel signing.

GRUB can become a nightmare very quickly. Currently I am on systemd-boot + ext4 for the boot drive. Has been working without any major issues. But boot drive backup with rsnapshot is very underwhelming

> Timeshift is similar to applications like rsnapshot, BackInTime and TimeVault but with different goals. It is designed to protect only system files and settings. User files such as documents, pictures and music are excluded.

On the other hand, a quick search looking for "that zfs based time machine thing" did reveal a new (to me) project that looks very interesting:

https://github.com/kimono-koans/httm

The root partition / and the home partition /home are different.

There's a /home/etc/ folder with a very small set of configuration files I want to save, everything else is nuked on reinstall.

When I do a reinstall, the root partition is formatted, the /home partition is not.

This allows me to test different distros and not be tied to any particular distro or any particular backup tool, if I test a distro and I don't like it, then it is very easy to change it.

- their package management tooling

- their filesystem layout (eg where do libraries etc go)

The gnome extensions need to be reinstalled, then they work and their configuration is preserved.

It must be because I don't use KDE now. I used it back in the day, when it was KDE 1.5 and 2.

It is a backup directory owned by root. The reason is that it sits in the partition that is preserved, and it is outside my user folder, because it is better organized that way.

[ /home/etc = ~/etc ] || echo theyre different

I may have had only one update that went wrong in 30 years of using Linux and that was just a bug introduced by a gfx driver in a new minor kernel version. I downgraded it and waited for the bug to be fixed upstream and that was it.

Folks who have run into issues, what was the root cause?

Currently the local folder is a samba mount so it's off-site.

The only tip I'd have for people using Borg is to verify your backups frequently. It can get corrupted without much warning. Also if you want quick and somewhat easy monitoring of backups being created you can use webmin to watch for the modifications in the backup folder and send an email if there isn't a backup being sent in a while. Similarly, you can regularly scan the Borg repo and send email in case of failures for manual investigation.

This is low tech, at least lower tech than elastic stack or promstack, but it gets the job done.

I remember working in a company that had a robot WORM system. It would grab a disc, it would be processed, take it out, place it among the archives. If a restore as needed the robot would find the backup, and read off the data.

I never worked directly on the system, and I seem to remember there was a window that the system could keep track of (naturally) but older disks were stored off site somewhere for however long that window was.

(Everything was replicated to a fully 100% duplicate system geographically highly separated from the production system.

I also tried to update it, when the graphical shell displayed a message saying that update is available. Of course, it bricked the system.

I've switched from Ubuntu to Mint to Debian to Fedora to Arch to Manjaro for personal use and had to support a much wider variety of distributions professionally. My experience so far has been that upgrades inevitably damage the system. Most don't survive even a single upgrade. Arch-like systems survive several major package upgrades, but also start falling apart with time. Every few years enough problems accumulate that merit either a complete overhaul or just starting from scratch.

With this lesson learned, I don't try to work with backups for my own systems. When the inevitable happens, I try to push forward to the next iteration, and if some things to be lost, then so be it. To complement this, I try to make the personal data as small and as simple to replicate and to modify moving forward as possible. I.e. I would rule against using filesystem snapshots in favor of storing the file contents. I wouldn't use symbolic links (in that kind of data) because they can either break or not be supported in the archive tool. I wouldn't rely on file ownership or permissions (god forbid ACLs!) Try to remove as much of a "formatting" information as possible... so I end up with either text files or images.

This is not to discourage someone from building automated systems that can preserve much richer assembly of data. And for some data my approach would simply be impossible due to requirements. But, on a personal level... I think it's less of a software problem and more of a strategy about how not to accumulate data that's easy to lose.

https://www.geeksmatrix.com/2024/07/linux-mint-220-whats-new...

fixing this should prove profitable

Saved my ass a few times.

On top of that BiT supports network backups and multiple profiles. I'm using it on my desktop systems with multiple profiles for years and it's very reliable.

However it's a GUI first application, so for server applications Borg is a much better choice.

Random notes/suggestions

- rsync is not a snapthot tool, so while in most of the cases we can rsync a live volume issueless on a desktop it's not a good idea doing so

- zfs support in 2024 is a must, btrfs honestly is the proof of how NOT to manage storage, like stratis

- it seems not much a backup tool, witch is perfectly fine but since the target seems to be end users not too much IT literate it should be stated clear...

I know it won't have the atomicity of a CoW fs, but I'd be fine with that, as the important files on my systems aren't often modified, especially during a backup - I'd configure it to disable the systemd timers while the backup process is running.

(I’ve only done this on VMs that could be paused before the snapshot, so YMMV.)

Sounds like it works similarly to git fork on GitHub? That is, if no files have changed, the snapshot doesn't take up any extra room?

It would make sense to hardlink a directory if everything in that tree was unchanged, but no filesystem will allow hardlinking a directory due to the risk of creating a loop (hardlinking to a parent directory), so directories are always created new and all files in the tree get their own hardlink.

Apple's Time Machine was given an exception in their filesystem to allow it, since they have control over it and can ensure no such loops are created. So it doesn't have that penalty creating hardlinks for every single individual file every time.

must have been this one :-D thanks for finding it!