AI scrapers are pretty widely ignoring robots.txt, and plenty lie about their user agents. https://rknight.me/blog/perplexity-ai-is-lying-about-its-use...

I'd fully expect OpenAI to do some checks that their bot isn't getting different responses than a seemingly real request.