I use it regularly. There probably are those communities you are talking about, but I haven’t personally seen them, so my advice would be to ignore them and join those that more align with what you’d like to see.
I’ve found many nice and welcoming communities on matrix.
I’ve found many nice and welcoming communities on matrix.