> That might work for something small like a curated collection of a few hundred sites.

Probably more like a few million but otherwise 100% true. Once you really need to scale you have to start losing some accuracy or correctness.

It helps that the goal of a search engine is not to find all the results but instead delight the user by finding the things they want.