I feel like even if it is good enough it’s only a matter of time until it lags and you lose a finger. Lulling us into a false sense of security is the real danger.
I don’t know about you, but it’ll be obvious to me in much less than 1ms when the passthrough is lagging. You know your hand moves, so when you don’t see it move it’s incredibly jarring.
You might be special. (I'm not being snarky.) I'm a digital musician. 5 ms lag is just barely perceptible to most of us.
My intuition regarding proprioception is similar to yours, though -- give me laggy input of my own hands and I'm still pretty likely to get things right.
Sound travels around 1.7 meters in 5ms. Acoustic musicians can play fine while being apart more of that distance, although there is a tendency to slow down, unless consciously keeping the tempo up (everyone individually feel that they are rushing a little bit, but that's how they just keep the pace).
There is a huge difference between consistent latency and jitter, but even then I doubt that 1ms latency + 1ms jitter would be very noticeable.
