Enter your domain. Your robots.txt is parsed to spec and evaluated per AI agent. The most common mistake is blocking the wrong bot — GPTBot and OAI-SearchBot are not the same thing.
Frequently Asked Questions
What is the difference between GPTBot and OAI-SearchBot?
GPTBot collects content for model training; OAI-SearchBot feeds the ChatGPT search index. For visibility, the critical agents are OAI-SearchBot and ChatGPT-User. Blocking GPTBot while allowing those two is a coherent choice for brands that do not want their content in training but do want to appear in search.
I have no robots.txt — is that bad?
Not directly: with no file, everything is considered crawlable. But you cannot declare your sitemap, and you cannot keep areas like an admin panel out of the index. A simple file solves both.
Should I remove the Crawl-delay line?
In most cases yes. Google ignores the directive; Bing and Yandex honour it. Because Bing is the index ChatGPT search depends on, keeping the line throttles your own indexing. If server load is a real problem, rate limiting belongs on the server side.
Track and optimize your brand visibility in AI search engines.