That's ok. You can talk to USB devices from userspace without issues. That's how many game controllers work. Also most (all?) SDR devices.
In most systems that impacts the mininal latency you can achieve though, so unless you have a system designed around it, you typically don't want real-time usage crossing to userspace too many times.