> Except the fiist thing openai does is read robots.txt. Then they should see th... | Hacker News

Hacker News new | past | comments | ask | show | jobs | submit

login

flutas on April 11, 2024 | parent | context | favorite | on: Anyone got a contact at OpenAI. They have a spider...

> Except the fiist thing openai does is read robots.txt.

Then they should see the "Disallow: /" line, which means they shouldn't crawl any links on the page (because even the homepage is disallowed). Which means they wouldn't follow any of the links to other subdomains.

niutech on April 12, 2024 [–]

This robots.txt has Disallow rule commented out:

    # buzz off
    #User-agent: GPTBot
    #Disallow: /

Join us for AI Startup School this June 16-17 in San Francisco!
Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact