robots.txt controls which crawlers may access which parts of a site. For AI visibility the critical detail is that training bots and search bots are separate: blocking the wrong one silently zeroes your visibility.
OpenAI runs three agents. GPTBot collects training data; OAI-SearchBot feeds the ChatGPT search index; ChatGPT-User fetches a page live when a user follows a link. A site that wants visibility must allow at least the last two.
The second common mistake is a Crawl-delay line. Google ignores the directive, Bing honours it — and Bing is the index ChatGPT search depends on. Keeping the line only throttles your own indexing.
Which bots decide visibility?
The most common mistake is confusing training controls with search crawlers. GPTBot, Google-Extended and Applebot-Extended govern model training; blocking them does not reduce search visibility.
Visibility is decided by different agents: OAI-SearchBot and ChatGPT-User for ChatGPT search, Bingbot for Copilot, Googlebot for AI Overviews, PerplexityBot and Perplexity-User for Perplexity, ClaudeBot for Claude. Declining training while wanting search visibility is a coherent position — but it requires blocking the right agent.
What does Crawl-delay actually do?
Google ignores the directive; Bing and Yandex honour it. Since Bing is the index behind ChatGPT search, leaving Crawl-delay in place slows down your own indexing.
If server load is genuinely a problem, the right place to solve it is server-side rate limiting, not robots.txt — there you can decide per agent and against real load.
Common misreadings of the file
The standard is misunderstood in three places. First, if a specific group exists for an agent, the User-agent: * group does not apply to it at all — groups do not merge. Second, when rules conflict within a group, the longest pattern wins. Third, an empty Disallow: line means "everything is allowed", not "nothing is".
Because of these three rules a file that looks correct to a human can be read very differently by a crawler. Checking it with a parser is safer than reading it.
Frequently asked
I have no robots.txt. Is that bad?
Not directly: with no file, everything is treated as crawlable. But you cannot declare your sitemap or close off sections like an admin panel. A simple file solves both.
If I block GPTBot, do I disappear from ChatGPT?
No. GPTBot is for training. Visibility in ChatGPT search is decided by OAI-SearchBot and ChatGPT-User. Blocking GPTBot while allowing those two is a coherent choice.
Is blocking all AI bots safe?
Safe, but it has a price: you will not appear in generative results at all. That can be a deliberate decision; done by accident, the loss is direct.
