I don't think distance is all that big a deal. Sure, you can't just drive there if it isn't near, but for most people the Olympics is a once-in-a-lifetime experience that they're willing to take a week off to visit. No different from any other trip, eg people will come to Paris from Australia just to see the Eiffel tower and the Notre Dame. Why wouldn't they fly to some other town to see the Olympics?

And let's face it: you have to be from a wealthy country to be able to do this in any case.