I don't completely disagree with your objection to the term, I think it is a term worthy of being critiqued. But, sometimes, when one is trying to communicate, the best option is to choose language which the majority of one's listeners will understand, even if one views that language as being in some ways flawed; choosing less flawed, yet less familiar, language may make it harder for the average listener to appreciate the point, and discussing the pros and cons of different terminological options can be a distracting detour which may also stand in the way of getting the primary point across. I think you are objecting, not to my point, but rather how I put it, because I think I could have made the same point without using that term. (And the same applies to my use of "culturally diverse", albeit for somewhat different reasons.)

The word you were looking for is neurodivergent. He was pointing out that you made a common linguistic error, and you barfed out a novel rather than investigate that you might've been wrong or misspoke.