noindex & X-Robots-Tag: The Tag That Hides You From AI Search
You can have perfect content, clean HTML, and every AI crawler allowed — and still be completely invisible. One small instruction, noindex, tells search engines and AI indexers to leave a page out entirely. It’s the most overlooked GEO problem because nothing on the page looks wrong.
This guide explains what noindex does, the two places it hides, and how to check whether one is quietly erasing you.
What noindex actually does
robots.txt controls whether a crawler may fetch a page. noindex is different: the crawler reads the page, then sees an instruction that says “don’t keep this in your index.” So the page is downloaded and understood — and then dropped. It won’t appear in Google’s results, and AI systems that build on a search index treat it the same way: not eligible to be surfaced or cited.
That’s why noindex is more dangerous than a robots.txt block. A block is loud and obvious. noindex is silent — the page loads fine for humans and just vanishes from search.
The two places it hides
noindex can be set in two ways, and you have to check both.
1. A meta tag in the HTML <head>:
<meta name="robots" content="noindex">2. An HTTP response header (X-Robots-Tag):
X-Robots-Tag: noindexThe header is the sneaky one. It isn’t in the HTML at all — it travels with the server’s response, so “View Source” won’t reveal it. A single line in a server config or CDN rule can apply noindex to a whole section of the site.
Why it’s so often left on by accident
- Staging carryover. Teams
noindexa staging or pre-launch site to keep it private, then forget to remove it when the site goes live. This is the classic one. - A CMS checkbox. Many platforms have a “Discourage search engines” toggle that quietly adds
noindexsite-wide. - A template default. One
noindexin a shared layout silently applies to every page that uses it. - A plugin or rule. SEO plugins and CDN rules can add an
X-Robots-Tagheader you never see in the markup.
How to check
- Meta tag: open “View Source” on the page and search for
noindex. If you find<meta name="robots" content="noindex">, that page is excluded. - Header: the header won’t show in the HTML. Check your browser’s Network tab (look at the response headers for the page), or run
curl -I https://yoursite.comand look for anX-Robots-Tagline.
If checking headers by hand sounds fiddly, that’s exactly the kind of thing an automated check catches in one pass.
When noindex is correct (don’t remove it blindly)
noindex isn’t always a mistake. You want it on pages that shouldn’t be in search: thank-you and confirmation pages, internal search results, admin areas, duplicate printer-friendly versions. The goal isn’t “never use noindex” — it’s making sure it isn’t sitting on the pages you actually want found.
The bottom line
Before worrying about advanced optimization, confirm your important pages are even allowed in the index. Check the <meta name="robots"> tag and the X-Robots-Tag header, remove any stray noindex from pages you want surfaced, and keep it only where you mean it. It’s one of the “let AI in” steps from our GEO overview — and the cheapest invisibility bug to fix once you find it.