I don't think there's any association with New York. etymonline claims that "downtown" originally refers to the idea that the suburbs are at a higher elevation than the city proper. I don't know how much stock I'd put in that, but I trust it more than someone's random intuition about the geography of New York.
It's the only available word for the concept in American English. I wouldn't recommend saying CBD; that would refer to cannabidiol or to marijuana in general.
I'm not especially impressed by the argument "if we assume it came from Manhattan, then we can know it came from Manhattan":
> Let’s suppose that Fosdick, incoherent from distress, forgot to include the words “to the” between “down” and “town” when describing the direction in which people were fleeing. With this in mind, we can assume that Lower Manhattan is in fact the original “downtown,” as posited by Schwarzer, and that the possible Bostonian origin of the word is simply a misunderstanding.
I'm also frustrated by the lack of sourcing here but the wiki article for downtown [1] also points to Manhattan, citing a book I don't have a copy of. Allegedly it was in use in Manhattan in the 1830s, which aligns with the date given by the etymology entry linked previously.
https://www.etymonline.com/word/downtown The word was in want of a closer look in fresh archives. On a closer search, it's older (at least 1780s), American, but not apparently a New York reference at first.
( https://www.etymonline.com/word/downtown : "The notion is originally literal, of suburbs built on heights around a city." )
It's the only available word for the concept in American English. I wouldn't recommend saying CBD; that would refer to cannabidiol or to marijuana in general.