Wouldn't having it in the same place every time make it more expensive/difficult for people to go see the games live? When the location changes every time, there will be a different set of people who go watch the games because it's nearby, but who possibly couldn't afford to (or wouldn't want to) travel to the other side of the world for them.
I don't think distance is all that big a deal. Sure, you can't just drive there if it isn't near, but for most people the Olympics is a once-in-a-lifetime experience that they're willing to take a week off to visit. No different from any other trip, eg people will come to Paris from Australia just to see the Eiffel tower and the Notre Dame. Why wouldn't they fly to some other town to see the Olympics?
And let's face it: you have to be from a wealthy country to be able to do this in any case.
Distance doesn't matter that much, as olympics in general are expensive enough to make transportation not the major part of the total price.
If they were held in my home town, that would be significantly cheaper because I wouldn't have to pay for accomodation; but if they were held in a city in a few hours driving distance, then it might as well be in a different continent, the tickets+hotel+food+driving will be just a bit cheaper than tikcets+hotel+food+planes.
What if we hosted it on a ginormous floating barge? If it was that big it could possibly be stable and not be subject to the movement of the ocean. Then to solve the issue of equal average location you just float it around the world. Heck, you could road trip it around the coasts.
