Maybe this doesn’t happen to you but: my infuse app on my Apple TV constantly clears the cache because it gets full. This requires it to rescan my library before I have access to my media. Kind of a pain. Server side processing like with jelly fin would prevent this from happening. This is the reason I bounce back and forth between infuse and Plex on my Apple TV.
I also have this all the time. I think it stores the metadata in iCloud (if you configure it) so it does not rescan your media, but it does need to redownload the iCloud data. No other app suffers from this, so I think it’s a bug.
I looked at Jellyfin about a year ago and settled on just playing media from my NAS using Infuse but maybe I need to take another look