Then why did Japan face a massive housing bubble in the 1980s, despite low immigration? And why did their housing problem only go away when they reformed their zoning laws to allow people to build? Population is a negligible factor. Laws restricting land use are artificially constraining the supply of housing, and these laws have grown more common and more popular.
This sounds like a bullshit post-hoc rationalization for xenophobia. There's no valid reason to deny entry to any and all immigrants who are willing to work and have clean criminal records. Which side of an imaginary line someone is born on shouldn't determine whether they are doomed to poverty and oppression.