I agree. If you're the only user of your instance, they can easily track you server-side by your IP address. You could mitigate this by routing your egress traffic (either from the individual device or network-wide, from your router) through a VPN.
It’s really not that much extra work to automatically rotate VPS IPs or VPN relays on an hourly or daily interval. Since most Invidious/Nitter guides don’t recommend doing this I guess it’s not very common.
Just remember that you’re still technically trackable via behavioral, configuration, or latency fingerprinting. Anonymity is a full-time academic endeavor, not a shell script one-liner.