I agree with your premise and I think there are two main reasons for this behavior.

Firstly even to migrate you have to have access to information and brain cycles to process it. Real poor are too busy to have that kind of time at hand.

Secondly even illegal immigration requires spending some money upfront and of course poor don't have it.

If you in the middle income range, it does make sense to move up the ladder by immigrating to a better place. If you are super rich, you can just buy citizenship.

Nothing is fair in life it seems.