The "fisheye distortion effect that plagues 3D views like this" that you describe is nothing of the sort. You are talking about a very wide angle of view. These images have a limited FOV, and it's not possible to zoom out so much that you get such distortion.
Actually it is not fisheye distortion that you describe at all - it is a rectilinear projection, and it's this type of projection that looks bad in a wide FOV. A fisheye projection looks much more natural if you are looking at a view with 110º FOV.
These things are not "distortion" at all but merely different projections of a spherical image on a flat surface. There is no "correct" way.
I realize that, just wasn't sure of how to describe it which is why I tacked on "effect" (might not be the actual type of distortion, but to a user the appearance is similar). From a user's perspective it most definitely does look wrong even though there is technically no "correct" way. Thanks for the link!
Actually it is not fisheye distortion that you describe at all - it is a rectilinear projection, and it's this type of projection that looks bad in a wide FOV. A fisheye projection looks much more natural if you are looking at a view with 110º FOV.
These things are not "distortion" at all but merely different projections of a spherical image on a flat surface. There is no "correct" way.
You can right-click on this panorama to change projection and understand more what I'm talking about. http://www.360cities.net/image/a-busy-morning