I'd love to see this happen. But i think that's very wishful thinking, outside of some very simple scenes - my R9 270 struggles to maintain framerate with some DK2 demos and the pi's VC4 isn't remotely in the same league.
I think we will need to look at streaming scenes from a more powerful machine rather than creating them on board. I don't believe we are ready in any way to process that much information on such a small system at this time (HoloLens also gives out quite a lot of heat from its on board processor). The old pi had trouble streaming higher bitrates, though then we still have the same ethernet so I am not sure how much is possible.