Hacker News
new
|
past
|
comments
|
ask
|
show
|
jobs
|
submit
login
hinkley
4 months ago
|
parent
|
context
|
favorite
| on:
I was wrong about robots.txt
All our customers were in North American but we let a semi naughty bot from the UK scan us and I will never understand why. It was still sending us malformed URLs we purged from the site years ago. WTF.
Guidelines
|
FAQ
|
Lists
|
API
|
Security
|
Legal
|
Apply to YC
|
Contact
Search: