Because most of the good sites won't let you crawl them any more, unless you're ... | Hacker News

Hacker Newsnew | past | comments | ask | show | jobs | submit

		Animats 8 months ago \| parent \| context \| favorite \| on: Two upstart search engines are teaming up to take ... Because most of the good sites won't let you crawl them any more, unless you're Google.

immibis 8 months ago [–]

Crawl them anyway. Adversarial interoperability is the name of the game.

Consider applying for YC's Fall 2025 batch! Applications are open till Aug 4
Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact