If you study, you may find the right to travel was already litigated at USSC. The gov't will not defend that right when most everyone signs it away by voluntarily applying for a license which has rules and fees. Gov't funded schools obviously don't teach this.

For extra credit, research what requires you to have an SSN. Hint - SSN is not part of passport application. Spoiler - your parents sold you out for the IRS tax deduction.