Resemblance to reality as UI design smell... Just because we have head tracking and stereo display, doesn't mean a UI should at all resemble reality. Years ago, when cell phones were new, and UI design was focused on onboarding novice users in an untrained culture, there was the idea of skeuomorphic ui design - the app should resemble the familiar physical thing. The calendar app should have lined pages, that flip around a spiral binding, surrounded by a textured leather margin. We don't do that anymore. It would be silly. As is 3D UI's resembling "spear chucking" reality. So yes, tossing xterms in the air, and 3D UI's that resemble VR games, are both common and bad ideas. Quietly assuming VR-gaming-like things cover the entire 3D UI design space, and drawing conclusions built on that assumption, is another common and bad idea.
> Splitting information across multiple depths is harmful
Frequently jumping between dissimilar depths is harmful. Less frequent, sliding, and similar depths, can be wonderful, allowing much denser and easily accessible presentation of information. I so wish my laptop had a native 3D display... DIY anaglyph and shutter glasses are a pain.
A general takeaway is... Much current commentary about "VR", is coming from a community focused on a particular niche, current VR gaming. One with particular and severe, constraints and priorities... that very don't characterize the entirety of a much larger design space. And that community, and commentary, has been really bad at maintaining awareness of that. So it's on you.
And my own stuff was so very limited and crufty, before I rather burned out on the yak shaving of it all. Though two requests for demos just this evening... sigh.
Perhaps picture the cylindrical "big screen" someone here mentioned. Run 3D programs like CAD in some of the windows. Control the window manger with keystrokes. Use a trackpad, a mouse, a 3D mouse. These inputs aren't tightly coupled to the rendered 3D space. Add a 6DOF controller. There's still no need to introduce a tight coupling - it can sit beside your mouse, or on your knee, and twitch. Discard the gaming objective of immersion, and show the outside world. Use a non-standard VR rendering stack, or AR, so that's performant. Open up the HMD horse blinders. Add some "comfort mode" tricks, and now the outside world is being used for balance, and the rendered world is freed to be bizarre. When your head moves, move the cylinder twice as fast, to reduce the need for head motion. Turn it with keys or mouse and no head motion. It's not a real cylinder - there's no reason when you turn it and turn it back, the same thing has to be there - it can be whatever you want, whatever is helpful. The motivation for a surface is coplanarity for optical scanning, and living in some hardware/wetware sweet spot. But one is a task-specific constraint, and both are local to the region currently shown, not global constraints. So punt the cylinder, and use an arbitrary aphysical topology. So now, while do you have a 3D visual display, and 3D HIDs, the contents and inputs are no more constrained to some imagined reality than in any usual window manager an apps. Choosing how much resemblance to reality to exhibit, as with scrolling "physics", is simply a UI design choice. ... Ah well - it's late - FWIW.
Not OP but the team at Leap Motion [1] has some very interesting demos. 3D UI's re-imagined from the ground up. Weirdly, because now everything is 3D, the best you can do is try to mimic actually 3D things perhaps, something that didn't work in 2D at all?
They're fun demos. And perhaps from the perspective of say mobile, they're kind of reasonable. Rotating your wrist? Waving your arm to press a button? Waving your arm to move a window? Not all that odd when holding a phone.
But from the perspective of laptops/desktops? What comes to my mind is: RSI; slowwwww, but maybe it's the touchpad version of a standing desk?; people use tiling window managers in part because waving a finger to place windows in 2D is already too burdensome.
I've just now failed again to find nice video, but if you've ever seen it, picture a professional artist tooling along on a Cintiq at insane speed. Stylus, and two handed multitouch flying. Zipping through enormous menus. Twitch gesturing through radial menus. Stylus input with pressure and tilt and rotation. Now imagine that interactive area was several centimeters high. And imagine transparent layers of content above the screen (you might sort of get a feel, by moving a semi-transparent window in front of another, and switch between reading one and the other, back and forth). Imagine a 3D visual environment extending below and above the screen. And around it, throughout the room. Now imagine your keyboard does multitouch, and instead of art, that's your code.
Resemblance to reality as UI design smell... Just because we have head tracking and stereo display, doesn't mean a UI should at all resemble reality. Years ago, when cell phones were new, and UI design was focused on onboarding novice users in an untrained culture, there was the idea of skeuomorphic ui design - the app should resemble the familiar physical thing. The calendar app should have lined pages, that flip around a spiral binding, surrounded by a textured leather margin. We don't do that anymore. It would be silly. As is 3D UI's resembling "spear chucking" reality. So yes, tossing xterms in the air, and 3D UI's that resemble VR games, are both common and bad ideas. Quietly assuming VR-gaming-like things cover the entire 3D UI design space, and drawing conclusions built on that assumption, is another common and bad idea.
> Splitting information across multiple depths is harmful
Frequently jumping between dissimilar depths is harmful. Less frequent, sliding, and similar depths, can be wonderful, allowing much denser and easily accessible presentation of information. I so wish my laptop had a native 3D display... DIY anaglyph and shutter glasses are a pain.
A general takeaway is... Much current commentary about "VR", is coming from a community focused on a particular niche, current VR gaming. One with particular and severe, constraints and priorities... that very don't characterize the entirety of a much larger design space. And that community, and commentary, has been really bad at maintaining awareness of that. So it's on you.