Of course it's your choice to make. Is someone forcing you to respond to request... | Hacker News

Hacker Newsnew | past | comments | ask | show | jobs | submit

jMyles on April 27, 2024 | parent | context | favorite | on: So you want to scrape like the big boys (2021)

Of course it's your choice to make.

Is someone forcing you to respond to requests you'd prefer to ignore?

theamk on April 27, 2024 | [–]

yes, people like OP who get the farms of scrapers.

The website owners make their preferences clear with robots.txt, IP blocks and other antibot technology. Scrapers intentionally ignore owners' desires and force the to respond.

paulryanrogers on April 27, 2024 | [–]

If crawlers are stealth DDoSing my site then I lose the ability to respond entirely.

Consider applying for YC's Fall 2025 batch! Applications are open till Aug 4
Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact