You have a point. In finance especially, I think being middle class is about the worst position to be in. If you're poor, you get into finance because you're fearless and hungry and worked your ass off. You work on the trading floor and get along fine.
If you're rich, it's like the OP said -- you get into finance because you have casual confidence and connections. You probably already know your boss or some of his colleagues socially, or your dad belongs to the same yacht club. And your boss likes you because you're just like him.
If you're middle class, though, you aren't hungry because you were actually pretty comfortable growing up, and you don't have the connections because your dad is an accountant at a mid-sized manufacturing company and doesn't have a yacht. So you're screwed.