Hacker News
new
|
past
|
comments
|
ask
|
show
|
jobs
|
submit
login
brianwawok
11 months ago
|
parent
|
context
|
favorite
| on:
Ban me at the IP level if you don't like me
I doubt that’s on purpose. The bad guys that don’t follow robots don’t bother downloading it.
Never assume malice what can be attributed to incompetence.
aequitas
11 months ago
|
next
[–]
Bad guys might download the robots.txt to find out the stuff they don't want them to crawl.
cyanydeez
11 months ago
|
prev
[–]
Its likely just a shitty attempt to rate limit bots
Consider applying for YC's Fall 2026 batch!
Applications
are open till July 27.
Guidelines
|
FAQ
|
Lists
|
API
|
Security
|
Legal
|
Apply to YC
|
Contact
Search:
Never assume malice what can be attributed to incompetence.