I had a copy of some Merleau-Ponty work in which this experiment was commented, IIRC the experimenter did really think his brain was damaged when the world appeared upside down to him without the glasses.

Don't we all actually go through this as babies? Where our mental image flips?

I don't understand why it would make any difference, from an evolutionary perspective, but apparently it does?