> Something is deeply wrong with such an adversarial ecosystem. If sites don't want to be found and indexed why go to any effort to include them?

I think it is not about being found. It is more about being copied.

These sites are afraid their content is stolen, so they only allow Google to crawl them.