I doubt that’s on purpose. The bad guys that don’t follow robots don’t bother do... | Hacker News

Hacker Newsnew | past | comments | ask | show | jobs | submit

		brianwawok 5 months ago \| parent \| context \| favorite \| on: Ban me at the IP level if you don't like me I doubt that’s on purpose. The bad guys that don’t follow robots don’t bother downloading it. Never assume malice what can be attributed to incompetence.

aequitas 5 months ago | [–]

Bad guys might download the robots.txt to find out the stuff they don't want them to crawl.

cyanydeez 5 months ago | [–]

Its likely just a shitty attempt to rate limit bots

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact