If the developer can get permission to build 3-5 story apartments on the lot, then their land suddenly becomes way more valuable than the single home ever was. Most retirees will gladly sell and live in luxury elsewhere. If a few don't, it won't make a difference.
If the developer can get permission to build 3-5 story apartments on the lot
wouldn't that only work for a single unit? Like, yes, if inside a single-family-zoned block a unit would get a special permission to be the only one to build high density, sure that piece of land would be more valuable than before.
But if the whole block gets rezoned, then there's a race to the bottom.
Many single family home owners will strenuously object to having a developer build a 5 storey apartment building next door because they don't want additional noise, traffic, and people looking down into their private back yard. It directly impacts their quality of life.