My understanding of the Constitution is that it sets limits on government power, e.g. it can't pass a law that restricts free speech. It does not enumerate all the things the federal government may do.
One of those limits (10th amendment) is all powers not enumerated by the constitution are left to the states or the people.
It’s ironic that one of the original grievances in the Declaration of Independence was the king restricting immigration to the colonies.
Freedom of movement is a common law right going back to the Magna Carta: It shall be lawful to any person, for the future, to go out of our kingdom, and to return, safely and securely, by land or by water, saving his allegiance to us, unless it be in time of war, for some short space, for the common good of the kingdom: excepting prisoners and outlaws, according to the laws of the land, and of the people of the nation at war against us, and Merchants who shall be treated as it is said above.
