llms.txt Is Either the Future of SEO or a Complete Waste of Time

This article will not tell you whether llms.txt will save your search rankings or bury them.

That question misses the point entirely.

The keyword llms.txt has sparked a debate that is already stale: SEOs measure it against ranking signals, dismiss it as irrelevant, and move on. But the real question is not whether it moves today’s metrics. It is whether you are building for the web that is coming or the one that is fading.

Here you’ll learn why the skeptics are measuring the wrong outcome, what llms.txt actually does, and why the smartest players are already treating it as infrastructure for the agentic web. The cost of implementing it is near zero. The cost of ignoring it might be invisibility.

What llms.txt Actually Does

llms.txt is a plain-text file placed at a domain’s root that lists the most important pages for large language models to reference. It is a recommendation layer, not a crawl directive or a ranking signal. The file tells AI models where to find your best content when they need to answer a question about your site.

The proposal, documented at the llms.txt standard page, aims to standardise how websites provide information to help LLMs use them at inference time. This is a subtle but critical distinction. Most SEOs hear “file at the root” and assume it functions like robots.txt, a control mechanism for crawlers. It does not.

Robots.txt tells crawlers what to avoid. Llms.txt tells models what to prefer. One is a gate. The other is a guide.

This misunderstanding explains the dismissal. SEOs test whether llms.txt changes crawl behavior or influences rankings. It does neither. It changes what an AI model considers authoritative when generating a response. That is a different outcome entirely, one that matters only if you believe AI-generated answers will shape how users find information.

The file itself is simple. A domain root. A list of URLs. A brief description for each. No syntax to learn. No schema to validate. The complexity is not in the file. It is in deciding what belongs there.

Why SEOs Are Measuring the Wrong Thing

The SEO community is evaluating llms.txt against ranking metrics it was never designed to influence. Does it improve citation frequency? Does Google read it? Does it correlate with traffic? These are the wrong questions.

llms.txt is a recommendation layer for AI models, not a ranking signal. The Ahrefs blog post calling it a speculative idea with no official adoption is technically correct, and completely misses the point. robots.txt had no official adoption when it first appeared either. It became infrastructure anyway.

The real question is whether llms.txt creates a machine-readable surface for AI agents. That is not a ranking question. It is a distribution question. When AI agents start relying on structured signals to decide which pages to cite, the sites that have them will be preferred. The sites that don’t will be invisible.

This is the same mistake SEOs made with structured data a decade ago. They asked whether schema markup moved rankings. It didn’t. But it changed how search engines understood content, and that changed everything. The smartest players treat AI visibility as three problems, crawl, understand, cite, and llms.txt solves the third one.

Measuring llms.txt against today’s search metrics is like measuring a foundation against paint color. You are looking at the wrong surface entirely.

The robots.txt Precedent Everyone Forgets

Robots.txt was dismissed as useless for rankings when it first appeared. SEOs ignored it because it did not influence search results directly. The same dismissal is happening now with llms.txt.

Before: SEOs treated robots.txt as a technical curiosity with no ranking value. They saw no traffic impact, no citation lift, no direct correlation with search performance. So they ignored it. The file sat on servers, unoptimised, unmaintained, and misunderstood.

After: Robots.txt became the infrastructure that made modern SEO possible. Crawl budget management, duplicate content control, site architecture signals, none of these existed as disciplines before robots.txt gave practitioners a machine-readable surface to control. The people who adopted it early built an advantage that took competitors years to match.

The parallel to llms.txt is exact. The file does not move rankings today. But it creates a machine-readable surface for the agentic web, the same kind of infrastructure play that robots.txt represented. The people who wait for proof of ranking impact will arrive late to a game already defined by those who saw the infrastructure first.

Who Is Actually Using It Right Now

The adoption numbers look terrible. A study of 300,000 domains found almost none using llms.txt. That is the wrong data point.

The question is not how many domains adopted it. The question is which ones did and why. Documentation sites were first. They live or die on whether AI tools cite their current content instead of stale forum posts. GitBook explains the logic clearly: llms.txt points models toward canonical docs and structured exports, not whatever random page an AI crawler happens to parse fastest. For a documentation team, that is not speculative. That is survival.

Ecommerce platforms are next. Yoast now generates llms.txt automatically for stores with large product catalogs. The logic is identical to documentation, when an AI agent asks what a store sells, the file gives it a curated list of products, collections, and policies instead of letting it guess from a sitemap. Stores with thousands of SKUs cannot afford an AI model citing last year’s discontinued line.

The pattern is clear. Early adopters are not chasing rankings. They are solving a specific problem: AI tools pull from whatever they find fastest. Give them better signals, and they cite current, authoritative pages. Give them nothing, and they guess.

This is where the smartest players separate from the pack. They are not measuring adoption rates. They are building for a distribution channel that does not exist yet, the same way early adopters of SERP intelligence analysis built advantages before competitors understood the game had changed. WryveAI’s approach to SERP intelligence follows the same logic: identify the signal before the crowd does, then act on it.

The adoption numbers will look better next year. By then, the question will not be whether you adopted llms.txt. It will be whether you adopted it early enough.

What Belongs in Your llms.txt File

The instinct is to dump everything. Every page, every product, every blog post. That instinct is wrong.

A curated content strategy is the only strategy that works here. The file is a filter, not a firehose. A bad llms.txt, one that lists irrelevant or low-value pages, trains AI models to ignore your site entirely.

Include only what you want AI agents to cite first and cite often.

• Canonical product pages
• Structured data exports
• XML sitemaps
• High-authority cornerstone content
• Core policy and about pages
• Top-tier category collections
• Current pricing or service pages

Every line in that file is a signal. The signal says: this page matters. The AI reads the file, trusts the curation, and surfaces those pages over everything else on your domain.

Volume is the enemy. A file with 200 entries is a file that will be ignored. Keep it under 20. Review it quarterly. Remove anything that no longer represents your best work or your current offerings.

The file itself is the curation. Make it count.

The Real Cost of Waiting

A single text file costs nothing to create. The cost of not creating it is measured in lost visibility when the agentic web matures.

This is not a ranking play. A cheat sheet for AI models does not move search results today. But AI agents are learning to trust curated signals over crawled noise. The sites that provide that signal will be preferred. The rest will be invisible.

First-mover advantage in infrastructure is hard to reverse. Early robots.txt adopters controlled crawl budgets while competitors watched their pages get ignored. The same dynamic applies here. When LLMs start defaulting to llms.txt as a trusted source, the sites that have it will be cited first. The sites that don’t will be an afterthought.

Waiting for official adoption is a luxury the market will not grant. The file takes fifteen minutes to create. The window to build that advantage closes the moment adoption becomes standard practice.

How to Create Your First llms.txt File

The whole process takes less time than most people spend deciding whether to start. Fifteen minutes from decision to deployment. No tools to buy. No permissions to request. Just a text file and a clear sense of what matters on your site.

Identify your most authoritative pages

Pull the ten to twenty pages that carry your site’s weight, the ones you would send a new client to first. Homepage, cornerstone content, product collections, policy pages. The goal is curation, not inventory. Every page you include is a signal. Every page you omit is a signal too.

Create a plain text file named llms.txt

Open any text editor, Notepad, VS Code, even the basic text app on your phone. Save the file exactly as llms.txt. No capital letters. No spaces. The filename is the contract. AI content writing tools and models look for this exact name at your domain root. Get it wrong and the file is invisible.

Step 3. List each page with a brief description on a separate line. One URL per line. Follow each with a short description of what the page contains. The description matters more than most people think, it tells the model why this page exists and when to cite it. Be specific. “Product catalog for winter collection” beats “Products page” every time.

Upload the file to your domain root

Use your hosting control panel, FTP client, or CMS file manager. Place the file at the root directory, the same level as robots.txt. If you can reach yoursite.com/llms.txt in a browser and see your list, the file is live. That is the only test that matters at this stage.

Test with an LLM tool to verify it reads the file

Open ChatGPT, Claude, or any model that supports web browsing. Ask it to read your llms.txt file. If it returns your curated list, the infrastructure works. If it returns nothing, check the file location and permissions. A file that cannot be read is a file that does not exist.

Fifteen minutes. That is the investment. The cost of not doing it is measured in missed citations and invisible pages when the agentic web matures. The file itself is the signal that you are building for what comes next.

The Infrastructure Question You Cannot Ignore

This entire debate frames the wrong question. llms.txt is not an SEO tactic to be measured against ranking reports.

It is infrastructure for the agentic web, a machine-readable surface that tells AI models where your authority lives. The skeptics are measuring citation frequency against search traffic and declaring the file worthless. They are checking the wrong dashboard.

The smartest players in this space are not chasing today’s algorithm. They are building for the distribution channel that is forming right now. Documentation sites and ecommerce platforms already understand this. They are not waiting for official adoption from every LLM provider. They are placing the signal now so it exists when the agents arrive.

That is the difference between building infrastructure and chasing tactics. One compounds. The other expires.

Create your llms.txt file today. Even if the list is imperfect. Even if only ten pages make the cut. The file itself is the signal, a declaration that your site understands how the next web works. The cost of waiting is not measured in time. It is measured in visibility you will never get back.

Frequently Asked Questions About llms.txt

What is an llms.txt file?

An llms.txt file is a plain-text document placed at the root of a website that lists the most important pages for large language models to reference when generating responses about your site. It acts as a curated recommendation layer, not a crawl directive, telling AI tools which pages carry the highest authority and relevance.

Does llms.txt actually work?

It works in the sense that major AI providers like OpenAI and Anthropic have publicly stated they will consider llms.txt files when sourcing information for model responses. The effectiveness depends entirely on whether the file is well-curated with high-signal pages rather than stuffed with every URL on the domain.

What is the difference between robots txt and llms txt?

Robots.txt tells search engine crawlers which parts of a site they are allowed or disallowed to access, functioning as an access-control file. Llms.txt does the opposite, it tells AI models which pages they should prioritize, serving as a recommendation file rather than a restriction file.

What should be in an llms.txt file?

The file should contain only your most authoritative and canonical pages, product collections, policy documents, structured data exports, and cornerstone content that represents your site’s best work. Every entry must earn its place through relevance and accuracy, because a bloated file trains AI models to ignore your signals entirely.