> Turns out that the BackBlaze filelist service (that watches for changes to files) is very poorly behaved. Initially we discovered that it fights with apple's mds
That's really disturbing, does it misbehave and interact with /dev/fsevents directly instead of using the public fsevents api? If so somebody needs to get flogged over that choice.
That's not really a question I have the knowledge to answer (though with some more guidance I'd be happy to look into it). I'm aware of fsevents, and I know for example that dropbox have their own dbfsevents so they can filter out what they need closer to the kernel. My understanding is that it gives you an api to be notified of changes to the fs. Maybe the BackBlaze case is complicated because they backup the whole filesystem by default?
The behaviour we saw was a constant scanning of all the metadata of all the files on the filesystem - not just things that were changing. It seemed that instead of being notified about changes it was relying on comparing to its cache by polling. There were huge folders with years of photos in that hadn't been touched in months being scanned again and again all day.
To be honest, my business partner contacted BB about it and they didn't seem as concerned as we were. It was at that point that for the first time in a couple of years I found myself with the desire to investigate backup services again. It's a shame because BB has been good when I've needed them. A drive died recently with a lot of important data on it (years of photos and music) and they had a replacement to me in a matter of days.
I'm still running their backups at the moment but I also started Arq running last night. It seems to fit my use-case perfectly.
Edit: I just ran it a again to check. The bzfilelist process does an lstat on every file in the system one by one.
> Edit: I just ran it a again to check. The bzfilelist process does an lstat on every file in the system one by one.
Wow, so it doesn't even listen for fsevents? What a terrible design, I understand running a full scan every now and then to ensure you haven't missed anything while the filesystem has been offline (in case it's mounted on another machine), but holy crap.
That's really disturbing, does it misbehave and interact with /dev/fsevents directly instead of using the public fsevents api? If so somebody needs to get flogged over that choice.