The implementation guides have reached consensus: add a file called llms.txt to your site and AI systems will understand your content better, cite you more accurately, and surface you in AI-generated answers. It's a 20-minute fix, the guides say, and every serious AEO practitioner should have one.
The evidence does not match that consensus.
llms.txt is a 2024 proposal by Jeremy Howard that has attracted a wave of tutorial content and tool support. But no major AI provider has publicly documented that they read and act on the file in production. Practitioners who implement it and track AI citation rates report negligible changes. The file is widely described as a standard; it is not yet one in any meaningful sense.
This post examines the evidence, explains why the category has avoided this conversation, and points to the AEO signals that demonstrably influence what AI systems retrieve and cite. The verdict is dated September 2026 — that matters, because this could change.
For the foundational layer of what AI answer engines actually retrieve, see the Answer Engine Optimization guide.
What llms.txt Is — and What It Claims to Do
llms.txt is a markdown-formatted file placed at the root of a domain (e.g., yourdomain.com/llms.txt). The spec, introduced by Jeremy Howard in 2024, proposes that the file should provide AI language models with a curated, human-readable summary of a site's content: what the site does, its key pages, and structured descriptions of important content.
The underlying logic is reasonable. AI crawlers ingest enormous amounts of content; a structured, author-curated summary could help them understand a site more accurately than inferring structure from a full crawl. In that sense, the proposal draws on the same intuition that made robots.txt useful.
The comparison to robots.txt, though, exposes the gap.
The Standard That Isn't (Yet)
robots.txt emerged from a 1994 proposal by Martijn Koster and was adopted almost universally within months because it solved a concrete problem crawlers and webmasters both needed solved. Every major crawler — Google, Bing, and every subsequent bot — checks robots.txt as a default step in the crawl protocol. The file works because the other side of the relationship committed to reading it.
llms.txt has no equivalent commitment. The spec is author-proposed and community-discussed. There is no standards body with provider sign-on, no crawl protocol requiring AI systems to check the file, and no enforcement mechanism analogous to the crawler ban that enforces robots.txt compliance.
The spec's own documentation acknowledges it as a proposal. That honesty deserves more weight than it gets in the implementation guides.
What the File Actually Contains
A well-formed llms.txt file includes a site description, a key links section with URLs and brief descriptions of important pages, and optionally a detailed content block with expanded markdown for specific high-value pages. The structure is clean and readable.
| Property | robots.txt | llms.txt |
|---|---|---|
| Introduced | 1994 (Koster proposal) | 2024 (Howard proposal) |
| Purpose | Crawler access control | LLM content guidance |
| Who reads it | All major crawlers (documented, universal) | Unknown — no provider documentation confirms |
| Adoption standard | De facto standard, universally enforced | Community proposal, no standards body |
| Enforcement | Crawlers that ignore it get blocked | None |
| Evidence it works | 30 years of documented crawl behavior | Absent at publication-scale |
The file format is not the problem. The question is whether anything on the other side is reading it.
For a grounding in what AI retrieval systems provably do read, the grounding queries post documents the observable retrieval layer: the actual search strings AI systems run before generating answers.
What the Evidence Actually Shows
The claim behind llms.txt adoption is that AI providers read the file and incorporate it into their retrieval and answer-generation processes. Evaluating that claim requires looking at what providers have documented.
Provider Documentation
As of September 2026, no major AI provider has published documentation confirming production-level llms.txt parsing. The absence is notable because these providers are not silent on AEO topics generally.
Google's documentation for appearing in AI Overviews is extensive: structured data schemas, E-E-A-T signals, content quality guidelines, and the role of grounding in Gemini's responses. llms.txt is not mentioned in any of these documents, in the Gemini developer documentation, or in Google Search Central's guidance for content optimized for AI systems. Anthropic publishes guidance for how Claude cites and retrieves content — llms.txt does not appear there either.
OpenAI's documentation for appearing in ChatGPT responses covers authoritative sourcing and structured content; no llms.txt reference. Perplexity has published more AEO-adjacent guidance than most providers, including notes on how its crawler processes content. No llms.txt documentation there either.
This absence does not confirm that the file is ignored. Providers don't document every signal they use. But it does mean there is no documented commitment — which is a meaningful distinction from the 30-year crawl commitment behind robots.txt.
Practitioner Reports
The qualitative pattern from practitioner communities is consistent: implementing llms.txt does not produce a measurable change in AI citation rates. This is not a universal claim — individual practitioners may see changes — but it is the modal reported experience.
The measurement problem makes this hard to nail down. AI citations are not directly attributable. A site that adds llms.txt in month one and sees more AI citations in month three cannot cleanly separate that change from content improvements, new backlinks, structured data additions, or natural variation in AI provider crawl patterns. The causal chain is opaque in ways that robots.txt is not.
What crawler log analysis qualitatively shows: AI crawlers do not consistently fetch the llms.txt path the way they consistently fetch robots.txt. Practitioners monitoring their server logs after implementing llms.txt report that while Googlebot, GPTBot, and Anthropic's Claude crawler fetch robots.txt on every crawl cycle, requests to llms.txt are sporadic and in some cases absent.
Documented AEO signals flow through confirmed retrieval paths. The llms.txt path remains uncertain — plausible, but not confirmed at the retrieval layer.
The Before/After Problem
A core limitation of the llms.txt evidence base is that the file's effects, if any, are diffuse and lagged. Unlike robots.txt — where blocking a bot produces an immediate, observable change in crawl behavior — llms.txt effects cannot be cleanly isolated.
This is not a theoretical concern. It means the practitioner community cannot build a meaningful evidence base for the file's effectiveness under current adoption conditions. The guides promoting llms.txt are not wrong to say it might help. They are wrong to imply it demonstrably does, because the evidence structure required to demonstrate that doesn't yet exist.
Why the Category Skips This Conversation
Understanding why every guide assumes llms.txt works helps practitioners read those guides more critically.
Implementation guides have no incentive to ask whether the file works. A tutorial that says "here's how to create and format llms.txt" is useful regardless of whether the file produces results. The author writes it, it ranks for "llms.txt how to," and readers implement the file. The guide's usefulness doesn't depend on the file's effectiveness.
Tool vendors who build llms.txt generators have a direct commercial interest in the file's perceived importance. The more essential llms.txt becomes in the practitioner's mental model, the more demand for generation, validation, and monitoring tools.
How "Best Practices" Solidify Before Evidence Does
The AEO category follows a well-established dynamic: a new optimization technique is proposed, tools are built around it, guides proliferate, and "best practices" solidify before evidence does. This is how most technical SEO requirements propagate — authority by repetition, not by verification.
llms.txt fits the pattern exactly. The file was proposed in 2024. Implementation guides appeared within weeks. Tool support followed. "You should add llms.txt" became conventional wisdom before any systematic adoption data was available.
This does not make llms.txt valueless. It makes the conventional wisdom premature.
What the Absence of Evidence Does (and Doesn't) Mean
"No documented evidence that llms.txt works" is not the same as "llms.txt definitely doesn't work." The two common interpretations are both too confident.
It's plausible that as AI providers mature their crawl protocols, llms.txt or a standardized successor becomes meaningful. The OpenAI ModelSpec and similar provider documents suggest providers are actively thinking about how to make AI systems more accurate and attributable — a problem llms.txt was designed to help with.
The honest position for September 2026 is: the file might help, in ways we can't yet measure, if providers are reading it in ways they haven't yet documented. That's a speculative upside, not a confirmed benefit.
What Actually Works for AI Search Visibility
Skepticism about llms.txt is useful only if it redirects toward the AEO signals that demonstrably influence AI retrieval. These are not theoretical — they operate at layers providers have documented.
Structured Data Schema
FAQPage, HowTo, BlogPosting, and Article schemas are read and cited directly by AI answer engines. Google's documentation for AI Overviews confirms that structured data is a factor in how Gemini generates and attributes responses. This is a documented signal, not a proposed one.
The implementation investment is real but bounded: adding FAQPage and BlogPosting schema to a post takes 30-60 minutes and produces machine-readable structured content that AI systems provably process. For most B2B sites, this is the single highest-ROI AEO investment available.
| AEO Tactic | Evidence of AI Adoption | Implementation Effort | Priority |
|---|---|---|---|
| FAQPage schema | Documented (Google AI Overviews guidance) | Medium — JSON-LD in page head | High |
| BlogPosting schema | Documented (Google structured data docs) | Medium — JSON-LD in page head | High |
| Grounding-query-matched content | Observable via Gemini API + OpenAI Responses API | High — content restructuring | High |
| Brand signal coverage (third-party) | Qualitatively documented via citation patterns | High — requires external presence | Medium |
| llms.txt | Not documented by any major provider | Low — one markdown file | Low |
Grounding Query Optimization
AI answer engines run background searches before generating responses. These grounding queries are the actual retrieval mechanism: the search strings that determine what content gets pulled into the context window before an answer is written.
Content that matches grounding query patterns gets retrieved and cited. Content that doesn't, doesn't — regardless of what llms.txt says about it. The grounding queries post documents how to observe these patterns using the Gemini API's groundingMetadata.webSearchQueries field and OpenAI's Responses API queries array. This is the layer where the evidence actually lives.
Brand Signal Coverage
AI systems draw on high-authority third-party mentions when constructing answers about a brand or topic. Coverage in industry publications, partner sites, and high-DA domains creates the brand signal layer that makes your site a trusted source — independent of how your llms.txt describes you.
A site with strong third-party coverage and weak llms.txt (or none) will consistently outperform a site with excellent llms.txt and thin external presence. The retrieval layer reads authority, not self-description.
Content Depth and Specificity
AI answer engines favor content that answers a specific question completely and accurately. A post that addresses "how does grounding query fan-out work?" in technical detail will consistently outrank a page that mentions the topic superficially, regardless of either page's llms.txt.
This is the oldest AEO signal and still the most durable: write content that is the best available answer to a specific question, and the retrieval layer will find it.
The Verdict on llms.txt in 2026
As of September 2026, llms.txt is not doing what the implementation guides imply — at least not at the documented scale those guides suggest.
The honest recommendation has three tiers:
If you want coverage and have an afternoon: Add the file. The format is well-defined, the implementation is low-effort, and there's no evidence of harm. If providers do adopt the spec at scale, you're covered.
If you're choosing between llms.txt and structured data / grounding-query-matched content: Do the latter first. These operate at the layer AI systems demonstrably read. llms.txt operates at a layer that may or may not be read, in ways that have not yet been documented by providers.
If you're tracking this space: Watch for provider documentation changes. The moment Google, OpenAI, or Anthropic publishes guidance that references llms.txt in their crawl or retrieval documentation, the calculus changes. That documentation does not exist as of this writing.
This verdict is dated September 2026. It may be wrong in a year. Document-based standards have surprised us before — and the problem llms.txt was designed to solve is real.
Take the GTM Maturity Assessment to see where your AEO and AI search visibility stack up against what's actually measurable.
Frequently Asked Questions
Does llms.txt actually work for AI search visibility?
No major AI provider has publicly committed to reading llms.txt in production. Practitioners who implement the file and track citation rates report negligible changes. The file is a well-designed proposal, not an adopted standard — which means its effects, if any, operate through an unconfirmed mechanism. The AEO signals that demonstrably influence AI retrieval are structured data schemas and grounding-query-matched content.
What is llms.txt and how is it different from robots.txt?
robots.txt is a 1994 crawler-control standard adopted universally across every major search engine — it works because every crawler checks it by default. llms.txt is a 2024 proposal that provides LLMs with a curated markdown summary of a site's content. The difference is adoption: robots.txt has 30 years of documented crawl behavior behind it; llms.txt has no equivalent commitment from any major provider.
Have any major AI providers officially adopted llms.txt?
As of September 2026, no major AI provider (Google, OpenAI, Anthropic, Perplexity) has published documentation confirming production-level llms.txt parsing. This doesn't confirm the file is ignored, but it does mean there's no documented commitment. Practitioner crawler log analyses suggest AI crawlers fetch the llms.txt path inconsistently, unlike robots.txt which is fetched reliably on each crawl cycle.
Should I add llms.txt to my B2B site?
Adding the file costs about 20 minutes and causes no harm, so there's no reason not to include it for baseline coverage. The priority question is whether to invest optimization time in llms.txt vs. FAQPage schema, grounding-query-matched content, and brand signal coverage — and the answer is clearly the latter first. Those operate at layers that AI systems provably read.
What AEO tactics should B2B marketers prioritize instead?
FAQPage and BlogPosting schema are the highest-ROI starting points — Google's own documentation confirms structured data is a factor in AI Overviews. After that: grounding-query-optimized content (structure your posts to match the search strings AI systems run before generating answers) and brand signal coverage on high-authority domains. These are documented inputs to the retrieval layer.
Is llms.txt worth the time investment in 2026?
For most B2B teams: implement it in an afternoon for coverage, then move on. The file is low-cost and low-risk. What it isn't is high-return — not yet, and not until providers document their adoption. Spend the real optimization time on structured data and content depth. Those produce measurable results under current retrieval architecture.
Sources
- Jeremy Howard, llms.txt (2024) — Original spec and rationale for the llms.txt proposal
- Google Developers, AI Overviews: Structured Data (2026) — Documentation confirming structured data as a factor in AI Overviews generation
- Martijn Koster, A Standard for Robot Exclusion (1994) — Original robots.txt proposal for historical comparison
- DemandLab, Grounding Queries: See the Searches AI Runs Before It Cites You — Observable retrieval layer documentation using Gemini API and OpenAI Responses API
- DemandLab, Answer Engine Optimization: The 2026 Complete Guide — AEO pillar covering structured data, schema implementation, and retrieval layer optimization
See How Your AEO Stack Measures Up
Most B2B sites are investing in AI visibility tactics that aren't measurable yet. The GTM Maturity Assessment shows where your AEO and pipeline systems actually stand — and what to prioritize first. Free, role-specific, takes 5 minutes.