robots.txt for AI Crawlers: Should You Allow GPTBot and ClaudeBot?
robots.txt is a small text file that tells crawlers which parts of your site they may read. AI crawlers respect it — so this one file can decide whether ChatGPT, Claude, and Perplexity are allowed to use your content at all.
What robots.txt does
It lives at the root of your site (yoursite.com/robots.txt) and lists rules for each crawler. Every crawler identifies itself with a “user-agent” name, and you can allow or disallow paths for each one.
The main AI crawlers
- GPTBot — OpenAI’s crawler.
- OAI-SearchBot — OpenAI’s crawler for ChatGPT search results.
- ChatGPT-User — fetches a page when a user asks ChatGPT to open a link.
- ClaudeBot — Anthropic’s crawler (Claude).
- PerplexityBot — Perplexity’s crawler.
- Google-Extended — controls whether Google may use your content for its AI products, separately from normal Google Search.
Allow or block?
- If you want to be cited in AI answers, allow them. Being read is how you show up as a source.
- Blocking is also a valid choice. Some publishers block AI crawlers to protect their content. That’s fine — as long as it’s deliberate. The real problem is blocking them by accident, such as a broad
Disallow: /left over from a staging site.
A copy-paste example (welcome AI)
User-agent: *
Allow: /
User-agent: GPTBot
Allow: /
User-agent: ClaudeBot
Allow: /
User-agent: PerplexityBot
Allow: /
Sitemap: https://yoursite.com/sitemap.xmlTo block one specific crawler instead, give it its own block:
User-agent: GPTBot
Disallow: /Common mistakes
- A leftover
Disallow: /that quietly blocks everyone. - Blocking a folder your content actually depends on.
- Forgetting the
Sitemap:line — it’s how crawlers discover all your pages.
The bottom line
robots.txt is the front gate, and one of the four layers in our GEO overview. Open yoursite.com/robots.txt, make sure AI crawlers aren’t blocked by accident, and add a Sitemap: line. It takes two minutes and decides whether AI can read you at all.