AI Crawler List: GPTBot, ClaudeBot, PerplexityBot, and How to Allow Them
AI companies read the web with named crawlers (each has a “user-agent” name). Knowing them lets you make a deliberate choice: allow the ones you want to be cited by, and block any you don’t. Here are the main ones as of this writing.
The main AI crawlers
- GPTBot (OpenAI) — OpenAI’s general crawler, used to gather content for its models.
- OAI-SearchBot (OpenAI) — fetches pages to surface and link them in ChatGPT’s search results.
- ChatGPT-User (OpenAI) — fetches a page on demand when a user asks ChatGPT to visit a specific link.
- ClaudeBot (Anthropic) — Anthropic’s crawler for Claude.
- PerplexityBot (Perplexity) — crawls pages so Perplexity can answer with and cite them.
- Google-Extended (Google) — not a separate crawler, but a
robots.txttoken that controls whether your content is used for Google’s generative AI (Gemini). Normal crawling is still done by Googlebot.
Microsoft’s Copilot largely draws on the Bing index, so the standard Bingbot matters there too.
How to allow them (robots.txt)
If you want to be readable and citable, the simplest rule is to allow everyone (the default if you have no robots.txt). To be explicit:
User-agent: GPTBot
Allow: /
User-agent: OAI-SearchBot
Allow: /
User-agent: ClaudeBot
Allow: /
User-agent: PerplexityBot
Allow: /How to block one
If you’d rather a specific crawler not use your content, disallow it by name:
User-agent: GPTBot
Disallow: /For Google’s generative-AI use specifically, add the token (this does not affect normal Google Search):
User-agent: Google-Extended
Disallow: /Blocking training crawlers is a legitimate choice — but be aware it can also reduce how often you’re read and cited by that company’s AI. Decide per crawler.
A caution: names change
AI crawlers are new, and their names and behaviors shift. Treat this as a starting point and check each company’s official documentation for the current user-agent and rules. A robots.txt typo — like accidentally disallowing / for everyone — can make you invisible, so verify after editing.
The bottom line
Know the crawlers, then choose deliberately: allow the AIs you want to be found in, block any you don’t, and double-check your robots.txt. For how robots.txt works in full, see robots.txt for AI crawlers, or the GEO overview for the big picture.