It could go down too, by 4%. Risk works both ways, hence the higher reward. You can't really expect anything, the only thing that you can do is trade risk for potential gain or loss and hope that you jump to the next ice-floe when you should.
Sure, on any given year it can swing wildly, but long term I believe that those who invest in individual stocks ("the rich") will do significantly better than inflation.
It sounds like you're assuming it's a fair game and that everyone gets to participate. The reality is most people don't get to do the things the rich get to do, and their gain tends to come at the everyone else's loss.
For example, the rich often pay a lower tax rate than the middle class due to various tax policies designed to benefit them, such as offsetting margin interest against capital gains or carrying forward capital losses indefinitely. Those gains have to be included in your overall returns and are why the rich are getting further ahead.
That all said, the best way to boost your wealth is to create it from scratch and start a company.
It sounds like you're assuming it's a fair game and that everyone gets to participate. The reality is most people don't get to do the things the rich get to do
Anybody can go to ETrade or Fidelity and buy an S&P index fund. (Or individual stocks, but that usually produces worse results).
carrying forward capital losses indefinitely
That may let "the rich" pay a low effective tax rate one year, but if so it means they paid a high effective rate in previous years where they had capital losses but couldn't use it to offset ordinary income. Getting rid of that offset would put an end to virtually all investing: investors would suffer 100% of their losses but keep only 85% or less of their gains, and that's not even accounting for inflation.
Miss the beat and you'll lose rather than gain.