Discoverability and smallness are at odds. This problem isn’t specific to the internet. That quaint, beautiful postcard town does not remain so once it’s been discovered. Eternal September happens everywhere.
Is it really true on the internet though? omg.lol could presumably stay "small-appearing" and "quaint" and have millions of users. How could you really tell the difference?
If it were all indexed you could drill down and find people who share your interests, that doesn't necessarily ruin the website, yeah?
For published works (say, a blog), discoverability is probably a good thing. For communities, however, with many-to-many communication (forums, etc.), discoverability is an antifeature. Community building requires some degree of common ground, which obscurity naturally filters for.
The other downside of mass-popularity is that above a certain scale, your community becomes a target. Both for individual bad actors (spammers, vandals, etc.) and for the apex predators of the small community world, commercial interests. Look at Maker Fair transitioning from a relatively niche convention of people showing off their cool stuff they made, and some miscellaneous sponsors and vendors looking to appeal to those people, to an over-commercialized affair with a thousand people trying to sell you a 3D printer, because that's the big moneymaker.
Community norms are what makes spaces worth inhabiting, and they just don't scale well.
