GEO Checker

robots.txt & llms.txt Generator

Create a robots.txt that lets AI assistants read your site, plus an llms.txt that points them to your best pages. Free, ready to save at your site root.

Generate your files

Fill in what you have. Only your real input is used — empty optional fields are left out.

One per line, as: Title | /path (or full URL)
Optional. Defaults to your site URL + /sitemap.xml

⚠ Watch out: Cloudflare can override your robots.txt

If your site is behind Cloudflare, its “AI Audit / Block AI bots / Content Signals” settings can block AI crawlers at the edge — before your own robots.txt is ever read. A perfect robots.txt does nothing if Cloudflare is quietly refusing GPTBot and ClaudeBot. If AI still cannot see your site after fixing robots.txt, check your Cloudflare bot settings and turn off AI blocking (or allow the crawlers you want).

Check a live site

Enter a URL to see whether its robots.txt welcomes AI crawlers and whether it has an llms.txt, using the same checks as our AI-friendliness scan.

What are robots.txt and llms.txt?

robots.txt is a plain-text file at the root of your site (yoursite.com/robots.txt) that tells crawlers which parts of your site they may read. AI companies run their own crawlers — GPTBot for ChatGPT, ClaudeBot for Claude, Google-Extended for Google’s AI, and others — and they obey robots.txt.

llms.txt is a newer, complementary file (yoursite.com/llms.txt) that gives AI a short, curated map of your site: a title, a one-line description, and links to your most important pages. It does not grant or deny access; it helps an AI find and understand your best content quickly.

Why this matters for AI visibility

If your robots.txt blocks or forgets the AI crawlers, your pages simply will not appear in AI answers — no matter how good they are. Many sites accidentally disallow these bots, or a template blocks everything that is not Google. Allowing them is the single most important switch for AI visibility.

Both files are checked by our AI-friendliness scanner: welcoming AI crawlers in robots.txt and publishing an llms.txt each contribute to your score. They are quick, invisible to visitors, and directly help AI assistants read and cite your site.

AI crawlers this allows

The generated robots.txt explicitly allows the major AI and AI-search crawlers:

  • OpenAI GPTBot GPTBot
  • OpenAI OAI-SearchBot OAI-SearchBot
  • OpenAI ChatGPT-User ChatGPT-User
  • Anthropic ClaudeBot ClaudeBot
  • Anthropic Claude-Web Claude-Web
  • Anthropic anthropic-ai anthropic-ai
  • Perplexity PerplexityBot PerplexityBot
  • Google Google-Extended Google-Extended
  • Apple Applebot-Extended Applebot-Extended
  • Common Crawl CCBot CCBot
  • ByteDance Bytespider Bytespider
  • Meta Meta-ExternalAgent Meta-ExternalAgent

How to use this tool

  1. Enter your site name, and optionally your URL, a one-line summary, and your main pages.
  2. Press Generate to get a robots.txt and an llms.txt.
  3. Save the robots.txt as /robots.txt at your site root.
  4. Save the llms.txt as /llms.txt at your site root.
  5. If your site uses Cloudflare, check its bot settings so AI blocking does not override your robots.txt (see the warning above).

Frequently asked questions

Is this generator free?
Yes. Generate robots.txt and llms.txt as often as you like, with no signup and no credit card.
Where do these files go?
Both live at the root of your site: yoursite.com/robots.txt and yoursite.com/llms.txt. They must be reachable at those exact paths.
Will allowing AI crawlers hurt my normal SEO?
No. Allowing AI crawlers such as GPTBot or Google-Extended is separate from Google’s normal search crawler. Your standard SEO is unaffected.
I allowed the crawlers but AI still cannot see my site. Why?
A common cause is a CDN or firewall blocking bots before robots.txt is read. In particular, Cloudflare’s AI-bot blocking can override your file at the edge — check those settings.