Silly question, but wouldn't putting something like Crawl-delay in the robots.tx... | Hacker News

Hacker Newsnew | past | comments | ask | show | jobs | submit

		lancer383 on March 12, 2015 \| parent \| context \| favorite \| on: The day I was DoSed by Google Silly question, but wouldn't putting something like Crawl-delay in the robots.txt file (somewhat) help alleviate the situation (if it is respected)? Or maybe even just block crawling of the entire site except for the homepage?

thekeywordgeek on March 12, 2015 [–]

Google doesn't respect crawl-delay, sadly. They rely on the Webmaster Tools setting, which is unavailable to me as I've described.

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact