You're not wrong. On the officer side, service academy seats and ROTC scholarships are competitive to get (like getting into any top-tier college), and then there's a decent amount of attrition along the path to commissioning, followed by more rounds of competitive selection and attrition for certain military communities (aviation, special operations, etc.), followed by the brutal up-or-out promotion and assignment system. So the people who are left standing after a while are usually fairly impressive.