If they ended the legacy system they could kiss all donations from millionaire and billionaire alumni goodbye. In the first place, this whole "racial diversity" thing is a diversion in order to maintain their legacy system. Can't accuse them of being elitist and exclusionary if they regularly admit a quota of non-legacy applicants deliberately selected for racial diversity.
Thus it's the downfall of their academic strength, weakening the collective intellect of their student body in two different ways (50% even?). Now, for new fields like Computer Science, 5 out of the 10 top US Comp-Sci universities are public and 8 out of 10 non-Ivy. It's even worse when you account for the fact that Ivy's like Columbia are egregious offenders of attempting to fraudulently boost their US News rankings and are soon going to be absent consequently.