Since 6 September, our AI Traffic feature has logged every visit to our own sites from a named AI crawler. Over the 36 days to 11 October, that came to 1,687 visits to our pages and files from nine AI crawlers, run by OpenAI, Anthropic, Perplexity, Meta, Amazon and Apple.
Between them, those crawlers fetched robots.txt 346 times and our sitemaps 96 times. OpenAI's search crawler alone checked robots.txt 229 times, and the training crawlers from OpenAI and Anthropic fetched the sitemaps 95 times. No AI crawler requested llms.txt, although our files were live throughout. The one on gensight.ai has been in place since June.
Only two requests for llms.txt appear anywhere in the log. They arrived on the evening of 11 October, within a second of each other, from Claude-User, and they were ours: a check, made through Claude while we wrote this piece, that the files were still up. So an assistant will fetch the file when a person tells it to. No crawler in our logs fetched it unprompted.
What llms.txt is
The idea was proposed by Jeremy Howard on 3 September 2024. It's a Markdown file at the root of a site, holding the site's name, a short summary and a curated list of links to the pages that matter, so that a language model can take in what a site is about without wading through navigation, scripts and adverts. Howard expected it to be useful mainly at inference, when an assistant is helping someone with a task, rather than in training. The proposal presents it as a complement to robots.txt: where robots.txt sets the rules for crawling, llms.txt adds context about the pages a model might want to read.
How it became a GEO staple
It spread through GEO advice quickly, and the reason isn't hard to find. A team can write one in an afternoon and point to it in a report, which is more than most AI visibility work allows. Developer documentation sites took it up, and OpenAI, Anthropic and Google all publish llms.txt files for their own developer docs, which did a lot of persuading on its own. Publishing a file isn't the same as reading other people's, though, and the proposal's own site doesn't name an AI system that consumes them.
Google Search has been the most direct about it. John Mueller wrote in June 2025 that "no AI system currently uses llms.txt", and Google's guide to AI features and your website says "you don't need to create new machine readable files, AI text files, or markup to appear in these features." The rest of Google is less consistent. Several of its developer sites publish llms.txt files, and Chrome's Lighthouse now checks for one in its experimental audits for agentic browsing, which is about AI agents finding their way around a site rather than about search.
We were part of the enthusiasm. The GenSight score has awarded a small number of points for a published llms.txt, our recommendations have suggested adding one, and our benchmark reports treated a missing file as a gap. In April, reviewing fifteen car manufacturers, we called the absence of llms.txt across the group a significant oversight. That gave the file more weight than the evidence for it ever supported.
What other people's logs show
Our sample is small: 36 days of logs from our own sites. On its own it proves very little, but it matches what larger studies have found this year. Ahrefs looked at 137,210 sites in its analytics data. 28% had an llms.txt file, which Ahrefs treats as an upper bound because its customers lean technical, and 97% of those files received no requests at all in May 2026. The bots that fetch pages to answer live questions barely touched them. EZY Research logged 83 sites over twelve weeks and found OpenAI's crawlers fetched robots.txt 3,990 times and llms.txt 7 times, while PerplexityBot never asked for it. Evil Martians measured their own site for two months and traced only 37 of about 770 fetches of llms.txt and llms-full.txt to named AI clients.
The one exception in EZY's data is Meta's crawler, which fetched llms.txt more often than robots.txt. It visited us 27 times in our window and never requested llms.txt.
Not relevant, or not yet?
Two readings fit the data, and we can't yet tell them apart. One is that llms.txt answers a question the AI companies aren't asking. Their crawlers already read HTML at enormous scale and their search systems already rank pages, and a summary written by the site owner is precisely the kind of self-description a cautious system might discount.
The other is that it's early. robots.txt itself ran as an informal convention for nearly three decades before it became a formal standard in 2022. If enough sites publish llms.txt, and AI agents that browse on someone's behalf find a tidy map of a site useful, someone may start reading it. This part is speculation.
What we're changing
We want our scoring to follow evidence, so we'll be reducing the weight llms.txt carries in the GenSight score. It will stay in our recommendations, but we'll be rewording it to say plainly what we know: there's little evidence today that AI systems read the file, so adding one is a cheap bet on how things might develop rather than a fix for visibility now. It costs little to keep, so we're not telling anyone to remove theirs. It will sit behind work with better evidence behind it: making sure the right crawlers are allowed in, and that the pages they fetch can be read without JavaScript and say something specific.
We'll keep watching the logs, and if the crawlers start asking for the file, we'll report that too.
Data: GenSight AI Traffic logs for our own sites, 6 September to 11 October 2026, counting visits from named AI crawlers to pages and files and excluding calls to our MCP connector endpoints. Sources: llmstxt.org; John Mueller on Bluesky (17 June 2025); Google Search Central, AI features and your website; Search Engine Journal, Google's llms.txt guidance depends on which product you ask (May 2026); Ahrefs, We analyzed 137K sites: 97% of llms.txt files never get read (June 2026); EZY Research, We put llms.txt on 83 websites. OpenAI read it 7 times (July 2026); Evil Martians, Which AI actually reads your site? (July 2026); RFC 9309, Robots Exclusion Protocol (2022).