llms.txt in 2026: do AI crawlers actually read it?

Mostly, no. As of mid-2026, the AI search and answer crawlers people most want to reach (GPTBot, ClaudeBot, PerplexityBot, and Google’s AI bots) almost never request your llms.txt. That is not a guess; it is what independent server logs keep showing, study after study. And yet the file is gaining ground in an unexpected place: Google’s own PageSpeed Insights now checks whether you have one. Both things are true, and this guide keeps them straight, so you know exactly what an llms.txt does and does not do for you today. If you want the how-to-write-one version first, that is the companion guide.

01 · The evidenceWhat the server logs actually show

The honest test is simple, and several teams have run it: drop a valid llms.txt at your root, then watch your access logs for known AI user agents and count who actually requests it. The results line up to an uncomfortable degree.

A few patterns repeat across all of them. Traditional crawlers like Googlebot and Bingbot touch the file occasionally. The dedicated answer-engine bots mostly do not, and in several logs never appear against the file at all. Meta’s and Microsoft’s AI crawlers are the most likely to show up, and even they tend to prefer structured files over plain text.

The takeaway is not that the file is broken. It is that, right now, the open-web answer engines are not its audience.

02 · The mixed signalWhat Google actually says about llms.txt

It depends which part of Google you ask, and in 2026 the two halves stopped agreeing.

Google Search is blunt. In a May 2026 optimisation guide it listed llms.txt among the tactics you do not need for its generative AI features, grouping it with other GEO folklore. Its search advocates had already said for months that Google does not use the file, with one likening it to the long-discredited keywords meta tag. No major AI lab (OpenAI, Anthropic, Meta, Mistral) has publicly committed to reading it in production either. The answer engines are steered by robots.txt, which they honour. llms.txt is not part of that contract.

Then, the same month, Google’s Chrome team pulled the other way. Lighthouse 13.3, the engine behind PageSpeed Insights, shipped a new Agentic Browsing audit category in its default configuration, and one of its checks is for a valid llms.txt: it verifies the file exists, is reachable without server errors, and is markdown with at least one H1, pointing site owners at the official spec. Because PageSpeed Insights inherits Lighthouse within a couple of weeks, every site owner who runs a report now gets flagged for a missing or malformed llms.txt, whether they wanted the signal or not.

So is Google for it or against it? Both, and coherently so once you separate the layers. Search ranking does not use llms.txt. Agentic browsing, where an AI agent reads and acts on your site, is a different surface, and that is where Lighthouse frames the file as a discoverability and efficiency signal: without it, its documentation notes, agents spend longer crawling your pages just to work out your structure. That distinction is the whole point of this article. The file’s future is the agent layer, not the search results page.

It is also a tell about where this is heading. Lighthouse bundles the llms.txt check with audits for WebMCP, the accessibility tree, and layout stability, which together read as a first attempt at scoring whether your site is built for AI agents at all. That bundle has a name now, agent readiness, and llms.txt is one row in it.

03 · The real use caseThe reader that does use it

Here is the part the doom takes miss. llms.txt has a genuine, working consumer today: the AI tools your technical users already run.

When a developer points Cursor, Claude Code, or GitHub Copilot at your documentation, those tools fetch llms.txt to load your docs as clean markdown instead of parsing navigation, scripts, and styling out of raw HTML. It keeps them inside their token budget and makes their suggestions about your product more accurate. This is why documentation platforms like Fern, and framework sites such as Next.js and Tailwind, now ship one as a matter of course. It is plumbing for AI-assisted development, not a search-ranking play.

That reframes the whole question. The audience for llms.txt is not, yet, the open-web answer engine. It is the assistant sitting in your user’s editor. If that user is a developer, the file is doing real work the moment you publish it. You can push it further with an instructions section, a pattern Stripe is often credited with, that tells assistants how to describe your product and steers them away from deprecated APIs. That is influence over what the tool says about you, which is the actual prize.

04 · The decisionSo should you ship one?

Yes, in almost every case, but go in with the right expectation. Find your row:

The throughline: ship llms.txt because it is cheap, harmless, useful to AI coding tools, and now flagged by Google’s own PageSpeed Insights, not because it will get you quoted in ChatGPT this quarter. And do not let it crowd out the work that actually drives AI visibility, which is the same work that drives everything else: pages a machine can parse, content worth citing, and enough presence that you actually get indexed and mentioned in the first place.

05 · VerificationHow to check whether anyone reads yours

The fastest validity check is now Google’s own: run your URL through PageSpeed Insights and read the Agentic Browsing section, which reports whether your llms.txt is present and well-formed. That tells you the file passes. It does not tell you anyone is reading it.

For who is actually fetching it, you need your own logs. Filter your CDN or server access logs for the AI user agents and count requests to /llms.txt and /llms-full.txt:

GPTBot              # OpenAI (crawl)
OAI-SearchBot       # OpenAI (search)
ClaudeBot           # Anthropic
PerplexityBot       # Perplexity
Google-Extended     # Google AI
Applebot-Extended   # Apple AI
Meta-ExternalAgent  # Meta AI

If you are on Cloudflare, its bot analytics breaks traffic down by user agent without you touching raw logs. For a sharper signal, drop a honeypot link inside the file pointing at a URL nothing else references. Any hit to that URL means something actually parsed the file and followed a link. Set your expectations low, watch over weeks rather than hours, and you will get an honest read on who, if anyone, is reading you.