You're obviously right that anyone with means will try to leave a high crime neighborhood. But "white flight" was absolutely a thing:

http://en.wikipedia.org/wiki/Blockbusting