The issue is your defining the community arbitrarily to be people in some city limit. Others including myself are defining community to be the homeless, workers who would like to be residents, but live outside the city limit, and employers who probably want more workers in commuting distance.
Unfortunately, there is only one San Francisco peninsula with cold air from the Pacific and warm air from the desert. I think the California legislature should over rule the city to deal with the homeless crisis and cost of living crisis. The Nimbys in San Francisco, like other cities, have no skin in the game to fix that problem as they benefit enormously from the crisis they make much worse.
We're not talking about skyscrapers, which have serious infrastructure requirements and need to be planned.
We're talking about low-rise apartment buildings.
Perhaps if I rephrased your statement as "If the community doesn't want poor people and minorities - you can't have poor people and minorities" you'd better understand why this is a morally bankrupt argument. Given the price of housing, it's effectively the same thing.
Your framing of 'I can do whatever I want' is what's 'ridiculous'.
You can't build a skyscraper on your property arbitrarily because it affects others in the immediate vicinity.
If the community doesn't want skyscrapers - you can't have skyscrapers.
There are infinity locations to build in the USA, which means can whatever you want on the whole.
Go away from the community that doesn't want the things you want to build, to one that does.