When most people think of the olympic games, they don't think about the IOC. They could care less about the IOC. They think about athletes, and sports, and national rivalries.
Most people would not give one fat hoot if the whole IOC got left in the dust to whither and die, while a new organization got set up to host similar events in one location.
If the IOC died, the worldwide championships of each sport would gain importance. Such championships are held in different cities every year, which makes me think that the IOC is just inheriting the behavior of moving cities. Probably 70% of the people just want to watch sports and, being a democracy, it's difficult to outweight their choice of spending public money on Olympics.
Most people would not give one fat hoot if the whole IOC got left in the dust to whither and die, while a new organization got set up to host similar events in one location.