llms.txt and AI crawlers: who is reading your site

Your site is being read by more AI systems than your analytics show, and the file most people reach for — llms.txt — is not the one that controls any of them. llms.txt is a proposed convention for handing LLMs a curated map of your site; adoption is real, but no major AI engine has confirmed it consumes the file. The crawlers that are definitely reading you today are governed by plain old robots.txt.

So the practical question splits in two: which bots are in your logs and what does each one feed, and which of them — if any — should you block. Get the second question wrong in one direction and your content trains models for free; get it wrong in the other and you silently vanish from the AI answers your buyers read. Here is the landscape as of mid-2026, hedged where it is still moving.

The AI crawlers actually hitting your logs

The operators document their own bots, and the documentation is worth trusting over folklore. The ones that matter:

| User agent | Operator | What it feeds | |---|---|---| | GPTBot | OpenAI | Training data for future models | | OAI-SearchBot | OpenAI | The search index behind ChatGPT search | | ChatGPT-User | OpenAI | Live page fetches when a user's chat needs your page | | ClaudeBot | Anthropic | Training data collection | | PerplexityBot | Perplexity | Perplexity's answer index | | Googlebot | Google | Google Search — including AI Overviews | | Google-Extended | Google | A robots.txt control token for Gemini training (not a separate crawler) | | CCBot | Common Crawl | The open web corpus many labs train on |

Three footnotes that keep this table honest:

• Google-Extended never appears in your logs. It is a token you address in robots.txt, honored by Google when deciding whether crawled content may train Gemini. The fetching is still done by regular Googlebot — and blocking Google-Extended has no effect on Search or on AI Overviews.
• The operators keep adding agents. As of mid-2026, Anthropic's docs also list user-triggered fetchers alongside ClaudeBot, and Perplexity documents a user-initiated agent alongside PerplexityBot. OpenAI's list lives at platform.openai.com/docs/bots. Check the source pages before you write rules; this table will age.
• CCBot is the sleeper. Common Crawl is not an AI company, but its corpus is a standard training ingredient across the industry. Blocking CCBot reaches more training pipelines than blocking any single lab's bot.

Training vs retrieval: the distinction that decides everything

Every AI crawler does one of two jobs, and they have opposite stakes for you.

Training crawlers (GPTBot, ClaudeBot, CCBot, the Google-Extended permission) collect content that shapes future model weights. Blocking them is a policy decision about whether your content should teach someone else's model. The visibility cost is indirect and slow — models learn about your category and your brand from what they can read — but nothing disappears from any product tomorrow.

Retrieval crawlers and live fetchers (OAI-SearchBot, ChatGPT-User, PerplexityBot, Googlebot) feed answers being composed right now. Block these and the consequence is immediate: your pages cannot be found, read, or cited at the moment a buyer asks. This is the difference that most blanket "block all AI bots" advice ignores.

The asymmetry matters because being cited at answer time is scarce and valuable. In our Brand Invisibility Report — 7,048 probes of five AI answer engines, May through July 2026 — tracked brands appeared in only about 2% of non-branded buying-intent answers, and Perplexity cited brand pages at the highest rate of any engine, 6.3% of probes. Those citations flow through exactly the crawlers a blanket block turns away. An empty shelf is an opportunity only if the engines can reach your pages; the mechanics of earning the citation are in how AI Overviews pick their sources and how to get your brand mentioned by ChatGPT.

llms.txt: what it is, and who actually reads it

llms.txt is a proposal by Jeremy Howard of Answer.AI, published in September 2024 at llmstxt.org. The idea: a markdown file at /llms.txt giving language models a curated summary of your site and links to your most important pages in a form that fits a context window — optionally with an expanded llms-full.txt containing the content itself.

Two honest statements about where it stands in 2026:

• Adoption is real. Documentation platforms generate it automatically, and a long list of developer-facing companies publish one. As a convention among docs sites and AI-adjacent tooling, it exists at meaningful scale.
• Consumption by major engines is not confirmed. Neither OpenAI, Anthropic, Google, nor Perplexity has stated that its products read llms.txt, and Google's John Mueller publicly compared it to the old keywords meta tag in 2025. Server logs across the industry rarely show the big crawlers fetching it. If someone tells you llms.txt is measurably driving their AI traffic, ask for the logs.

Our verdict: llms.txt is cheap to add, carries no real downside, and may become useful if consumption materializes — but it is a courtesy map, not a control surface. Treat it as optional polish. Robots.txt is where decisions get enforced.

Robots.txt patterns that match real intent

The most common sane policy for a commercial site: stay retrievable everywhere, opt out of training.

# Opt out of model training
User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: Google-Extended
Disallow: /

# Stay visible in AI answers (explicit, though allow is the default)
User-agent: OAI-SearchBot
Allow: /

User-agent: ChatGPT-User
Allow: /

User-agent: PerplexityBot
Allow: /

And the full lockout, for completeness — appropriate only when you have decided AI answer visibility is worth giving up:

# Block training AND answer-time retrieval
User-agent: GPTBot
Disallow: /

User-agent: OAI-SearchBot
Disallow: /

User-agent: ChatGPT-User
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: PerplexityBot
Disallow: /

User-agent: CCBot
Disallow: /

Remember what robots.txt is: a published request that reputable operators honor, not an enforcement mechanism. The major labs document compliance; anything less reputable ignores the file entirely.

When blocking makes sense — and when it silently costs you

Blocking is defensible when your content is the product. Paywalled journalism, original research you license, premium data — content whose whole value is exclusivity. A training block is also a negotiating position: several large publishers blocked first and struck licensing deals second.

Blocking retrieval is usually a silent self-inflicted loss for everyone else. If you sell products or services, AI answers are a discovery surface where — per the measurement above — almost nobody in your competitive set is present yet. Blocking OAI-SearchBot or PerplexityBot does not protect anything worth protecting; it removes you from answers your competitors will eventually fill. The failure mode is invisible: no error, no warning, just answer engines that never mention you and no way to notice unless you are measuring.

The block-everything default some CDNs and plugins now offer makes this worse: teams inherit a full AI lockout they never consciously chose. Audit what your edge layer is doing in your name.

Verify who is actually crawling: read your logs

Two reasons to look at access logs rather than trust assumptions: user-agent strings are trivially spoofed, and your robots.txt may not be doing what you think.

Start with a simple sweep over your access log:

grep -iE "gptbot|oai-searchbot|chatgpt-user|claudebot|perplexitybot|ccbot" access.log \
  | awk '{print $1, $12}' | sort | uniq -c | sort -rn | head -25

Then verify the traffic is genuine. OpenAI, Anthropic, Perplexity, and Google all publish the IP ranges their crawlers use — a claimed GPTBot hit from an address outside OpenAI's published ranges is an impostor, and worth blocking at the firewall rather than in robots.txt. If you sit behind a major CDN, its bot analytics will usually classify verified AI crawlers for you, which is the fastest audit of all.

While you are in the logs, check three things: whether retrieval bots are reaching your money pages, whether a stale disallow rule is turning them away, and whether anything is fetching /llms.txt at all — that last one tells you more about the standard's real adoption than any blog post, including this one.

FAQ

Does llms.txt replace robots.txt?

No — they do different jobs. Robots.txt is the access-control file that documented crawlers honor. llms.txt is a proposed content map for LLMs, with no confirmed consumption by major engines as of mid-2026. Publish llms.txt if you like; make your actual decisions in robots.txt.

Will blocking GPTBot remove my site from ChatGPT?

Not from answers. GPTBot only feeds training. ChatGPT's search answers come through OAI-SearchBot's index and ChatGPT-User's live fetches — block those two and you disappear from answer time; block GPTBot alone and you have only opted out of training future models.

Does blocking Google-Extended remove me from AI Overviews?

No. Google-Extended governs Gemini model training only. AI Overviews are fed by normal Googlebot indexing, and the only way out of them is standard search controls like nosnippet — which cost you regular search visibility too.

Should I add llms.txt to my site?

It costs a few minutes and risks nothing, so adding one is fine — a short summary plus links to your key pages. Just do not stop there and call it an AI strategy: measurement of whether engines mention and cite you is where the signal is, and our AI visibility tracking guide covers how to get it.

Want to know what the answer engines currently say about your brand? Maark's AI visibility measurement tracks mentions and citations across five engines daily — join the waitlist.