Hearing that VR works well with passthrough makes me want to try it again. I remember when I last tried it, everything worked great except for sound, which was staticky and choppy unless I handed the whole soundcard over to the VM.
Some setup is required, but KVM passing the system audio to pipewire-pulse worked first try for me actually. I'm normally using a virtual desktop to stream everything to my Oculus Quest 2, including audio, and it works surprisingly well.