Vincent JOSSE
Vincent is an SEO Expert who graduated from Polytechnique where he studied graph theory and machine learning applied to search engines.
LinkedIn Profile
No, most websites do not need an llms.txt file as a top AI SEO priority today. But if you can add a clean, accurate file in under an hour, it is reasonable to do it as a low-cost bet.
That is the practical answer. The expert answer is more nuanced because the signals are mixed.
On one hand, Google’s own AI optimization guidance explicitly lists llms.txt and special AI markup files under things to stop doing for Google. In other words, Google is saying: focus on unique content, crawlability, indexability, page experience, snippets, and technical fundamentals instead.
On the other hand, Chrome DevTools has a Lighthouse page explaining llms.txt for agentic browsing. That page describes llms.txt as an emerging convention for giving LLMs and AI agents a machine-readable Markdown summary of a website’s content. It also explains that the audit treats a 404 as Not Applicable because the file is optional, while server errors when requesting it can be flagged.
So the real question is not simply whether llms.txt is useful or useless. The better question is: where does it belong in your SEO priority stack?
What it is
An llms.txt file is a proposed text file placed at the root of a domain, for example:
The idea comes from the llms.txt emerging convention. It is meant to give AI systems a concise Markdown summary of your site, including key sections, important URLs, documentation, policies, and other canonical resources.
It is often described as being similar to robots.txt, but that comparison can be misleading. A robots.txt file gives crawling instructions to bots that choose to follow it. An llms.txt file is not a crawling control protocol. It does not block anything, grant special access, or guarantee that AI systems will use your content.
A good llms.txt file is closer to a machine-readable site guide. It says: here is what this site is about, here are the most important pages, and here is the content you should treat as canonical.
The Google split
The confusing part is that Google’s ecosystem is sending two different signals.
That last point is where the “do what I say, not what I do” question becomes fair.
Google Search Central documentation pages have View as Markdown and Copy as Markdown buttons. That does not prove that llms.txt is a ranking factor. It does not prove Googlebot is fetching your file. But it does show that Google understands something SEOs have been saying for a while: clean, extractable, Markdown-like content is convenient for machines, agents, developers, and retrieval systems.
The most likely explanation is not conspiracy. It is simply that Google Search and Chrome DevTools have different goals. Google Search Central is telling site owners what matters for Google Search. Chrome DevTools is documenting a broader agentic browsing convention that may be useful outside classic ranking systems.
The SEO reality
Google has often said not to do things that still work in practice.
Backlinks are the obvious example. Google’s guidance has long emphasized earning links naturally and avoiding links intended to manipulate rankings. In a perfect world, you would publish excellent content and high-quality websites would discover and link to it organically.
In practice, waiting for links to appear often does not work at all, especially for startups, SaaS companies, local businesses, affiliate sites, and new domains. A lot of SEO agencies buy links, exchange links, run partner link programs, and negotiate insertions, even though Google’s idealized version of link earning is much cleaner.
That does not mean every link tactic is safe or recommended. It means SEO teams have to separate three things:
What Google officially recommends
What search systems can actually reward
What creates unacceptable policy, brand, or long-term risk
This is the same practical mindset behind BlogSEO’s backlink feature, which automatically finds contextually relevant backlinks for your site without needing to spend hundreds of dollars on paid insertions or send thousands of cold emails for guest posting.
The lesson for llms.txt is similar, but with one major difference: unlike backlinks, there is currently weak evidence that llms.txt is being used at scale by major AI crawlers.
What logs show
The strongest argument against prioritizing llms.txt is not Google’s statement. It is server log evidence.
Multiple studies and public log-file analyses have reported little to no evidence that major LLM user agents or crawlers are consistently fetching llms.txt files from Apache, Nginx, or CDN access logs. In plain English: when people check their server logs, they often see AI crawlers hitting normal pages, but not requesting /llms.txt.
That matters because if a crawler never requests the file, the file cannot influence that crawler’s understanding of your site.
You can check your own logs with simple searches like these:
If you use Cloudflare, Fastly, Vercel, Netlify, or another edge platform, check those logs too. Server-side Apache logs may not capture every request if traffic is terminated at the CDN layer.
The current pattern is fairly clear: llms.txt is an interesting convention, but there is no strong public evidence that the major AI crawlers depend on it today.
Could that change? Absolutely. Agentic browsers, AI coding tools, research agents, and smaller crawlers may adopt it faster than major search engines. That is why dismissing it completely is probably too strong.
Real priorities
For most websites, llms.txt should sit below crawlability, indexation, content quality, internal linking, and authority building.
If your pages are not indexed, if Google cannot crawl your important URLs, if your blog has orphan pages, or if your internal links do not show which pages matter, an llms.txt file will not save you.
This is why, in most AI SEO audits, fixing crawlability and internal linking is more effective than adding an llms.txt file.
Internal links help crawlers discover pages, understand relationships, distribute authority, and identify canonical topic hubs. They matter to classic SEO and they also matter to AI retrieval systems that rely on crawlable web content. If you need a higher-impact technical improvement, start with scalable internal linking rather than a speculative file. BlogSEO has a full guide on how to rank with internal links that scale if that is your current bottleneck.
When to add it
You should consider adding llms.txt if it is cheap, accurate, and easy to maintain.
It makes the most sense for websites with documentation, knowledge bases, API references, technical guides, large blog archives, product education hubs, or lots of evergreen content. These are the types of sites where a concise machine-readable map can help a human, developer, or agent quickly understand what matters.
It is less important for a five-page brochure site, a local service business with basic pages, or a site that has bigger technical SEO issues.
A simple decision rule works well: if adding llms.txt takes less than one hour and does not distract from higher-priority SEO work, add it. If it requires a multi-week project, custom CMS work, legal review, and a new maintenance process, pause and fix your fundamentals first.
What to include
A practical llms.txt file should be short, honest, and stable. Do not turn it into a sitemap clone with thousands of URLs. Do not include private pages, paid-only resources, staging URLs, or anything you would not want copied into an AI system’s context window.
A basic version can look like this:
Keep the file clean. Use absolute URLs. Link to your most important hubs and canonical resources. Review it when your site architecture changes.
Also make sure the file returns a clean 200 status if you publish one. The Chrome DevTools documentation says the Lighthouse audit is Not Applicable when the file returns 404 because the file is optional, but it can flag server errors when attempting to retrieve it. In other words, having no file is fine. Having a broken file is worse.
What not to do
The biggest llms.txt mistake is treating it like a shortcut for AI visibility.
Do not add one and assume ChatGPT, Claude, Perplexity, Copilot, or Google AI Overviews will suddenly cite you. AI answer visibility is influenced by many factors, including crawlability, authority, content clarity, source reputation, entity consistency, freshness, and whether retrieval systems can access and trust your pages.
Also avoid stuffing the file with keyword variations. Google’s AI optimization guide warns against mass-producing pages for query variations and rewriting content specifically for AI. The same principle applies here. A bloated llms.txt file full of SEO phrasing is not helpful. It is just another artifact to maintain.
If you add the file, treat it as a navigation aid, not a ranking lever.
My take
My expert opinion: you do not need an llms.txt file for AI SEO today, but it usually does not hurt to add one.
It should not be a board-level SEO initiative. It should not replace technical SEO. It should not delay content production, internal linking, indexing fixes, or authority building. For most sites, those will produce a much higher return.
But if your CMS or developer can publish a concise root-level Markdown file quickly, it is a reasonable hedge. The convention is young. Agentic browsing is evolving. Chrome DevTools documenting it is not meaningless, even if Google Search Central says not to focus on it for Google Search.
The best position is pragmatic:
Do not expect direct rankings from
llms.txt.Do not prioritize it over crawlability or internal links.
Do add it if the cost is low and the file can stay accurate.
Do monitor your logs to see whether anyone actually requests it.
That is the kind of AI SEO work that makes sense in 2026: small bets on emerging standards, backed by serious investment in fundamentals that already work.
FAQ
Is llms.txt a Google ranking factor? No public evidence suggests that llms.txt is a Google ranking factor. Google’s AI optimization guidance specifically lists it as ineffective for Google, so you should not expect ranking gains from adding it.
Why does Chrome DevTools mention llms.txt if Google says it is not useful? Chrome DevTools documents llms.txt as an emerging convention for agentic browsing, not as a Google Search ranking requirement. The Lighthouse audit also treats a missing file as optional, while flagging server errors if the file is requested and broken.
Do AI crawlers fetch llms.txt? Public server-log studies have found little or no consistent evidence that major LLM crawlers fetch /llms.txt at scale. You should verify this on your own site by checking Apache, Nginx, CDN, or hosting logs.
Should every website create one? No. If your site has crawlability problems, weak internal links, duplicate content, or poor indexation, fix those first. Add llms.txt only if it is quick and easy to maintain.
What is the biggest risk of adding llms.txt? The main risk is not the file itself. The risk is creating a stale, inaccurate, or overstuffed file that gives teams a false sense of AI SEO progress while more important issues remain unresolved.
How often should I update it? Update it whenever your site architecture, product positioning, documentation, or core content hubs change. For most sites, a quarterly review is enough.
Scale what works
llms.txt is a useful small bet, but the compounding gains usually come from better content systems: keyword research, consistent publishing, internal linking, technical hygiene, competitor monitoring, and authority building.
BlogSEO helps automate that workflow with AI-powered content generation, auto-publishing, website structure analysis, keyword research, competitor monitoring, brand voice matching, internal linking automation, CMS integrations, scheduling, and contextual backlink discovery.
Start with the fundamentals that move traffic, then add emerging AI SEO signals where they make sense. You can try BlogSEO with a 3-day free trial or book a demo to see how the full system works.
