Except that studies show that (at least above some relatively low threshold), money spent on a campaign doesn't correlate with election success. There's no point in throwing out the First Amendment (and possibly the most important aspect of it at that) for something that's not going to make any difference.
If I contribute $2000 to a party or a politician, and Charles Koch contributes $2M, which one of us is going to get the ear of our representative when we call?
In my case, some intern will dutifully take my call. In Charles Koch's case, he will have that congressperson's direct phone line. When he goes to Washington, they'll have a sit down meal together. Hell, if he wanted it, he could have asked for and received a cabinet position or an ambassadorship.
That isn't just free speech -- that is derailing democracy. By limiting contribution amounts, it will make the elected less beholden to big donors.