AI Traffic is now in beta on the GenSight platform. It reports AI crawler activity observed at the edge of your site: which AI companies visit, what they fetch, how much of your site they have never read, and whether they honour the rules in your robots.txt. Setup takes about two minutes with a Cloudflare token, or a small snippet on any other host.
It exists because an audit and a traffic log answer different questions. An audit measures what AI engines are able to read on your site: structure, entity signals, schema, llms.txt. That is a measure of capability. AI Traffic is a record of events, and the two are most useful read side by side.
Four classes of visit
Traffic is separated by purpose rather than reported as a single "AI bots" figure, because a crawler gathering training text and a crawler fetching your page for a waiting user operate on different timescales and imply different work.
Training crawlers are the slow clock. What they take now shapes what models know about you in a year, and there is no way to hurry it. Live retrieval is the fast clock: an engine reading your page right now, because a user is waiting for an answer that may include you. AI search indexing sits between the two, building the answer-engine indexes that decide whether you are a candidate at all. And agent connections are the newest category, agents reaching your service directly through MCP or your API rather than reading pages at all.
The training and retrieval split corresponds to the memory-versus-retrieval behaviour we have written about previously: models answer from parametric memory when confident and retrieve live when they are not. AI Traffic makes that split observable from your own server data rather than inferred.
Crawl coverage
Crawl coverage compares your sitemap inventory against what has been fetched, and reports how much of your site each AI company has read. The distinction it measures is between crawlable and crawled.
It also lists the specific pages no AI has fetched in the last thirty days. In our testing this list frequently contains product pages, documentation and case studies, the material most relevant to a category question, while the homepage is fetched repeatedly. Those pages are addressable once identified.
Robots compliance
The compliance panel reports observed bot behaviour against your robots.txt rules, which are advisory rather than enforceable. It shows which bots respected your directives across checked fetches, and which fetched disallowed paths.
One distinction is applied deliberately: a fetch triggered by a real person asking an assistant to look at a page is not an automated crawler ignoring your rules. Vendors treat those on-demand fetches differently, and so does the panel. They are shown for awareness and not counted as violations.
This connects to something we found in our retail benchmark earlier this year, where the best-prepared high-street brand was serving a genuine llms.txt from behind a bot-wall that returned 404s to automated readers. Nobody in that company knew, because nothing they owned was watching the door. Observed traffic is how that gets caught.
Evidence, inside the audit
Audit signals can now carry live crawl evidence. Where an AI engine has been observed fetching a file, the signal is marked accordingly, so a finding can read as present, well formed, and observed being read, rather than present and well formed alone.
The llms.txt case is the one we are most curious about, and it is worth being straight about the state of the evidence. llms.txt is an emerging standard, and whether AI engines routinely fetch it is not settled. We recommend it because it is cheap, well formed, and plausibly useful, not because anyone has proven it changes outcomes. What AI Traffic adds is the ability to stop speculating: if engines are requesting the file on your site, you will see it in the log, and if they are not, you will see that too. Over time, across enough connected sites, that turns a standing argument in this field into a measurable question. We will publish what we find either way.
What the data is for
Three uses have come up immediately in our own testing.
Checking that roadmap work lands. Audit recommendations are hypotheses until something changes. With observed traffic you can watch the classes move after you ship: whether making a section more machine-readable is followed by training crawlers taking more of it, whether structural fixes coincide with more live retrieval, whether an API or MCP endpoint starts attracting agent connections at all. It closes the loop between fixing something and seeing an effect, which visibility work has historically lacked.
Putting your messaging where the attention is. Coverage data cuts both ways. If the pages carrying your positioning, your differentiators, your pricing logic, are not being fetched, you have two options and both are valid: make those pages more retrievable, or move the material onto the pages engines already read heavily. Most brands write for a crawl pattern they have never seen. This is the first time that pattern is visible.
Checking your own defences are not the problem. Enterprise bot protection blocks automated traffic by design, and AI crawlers are automated traffic. We found a national retailer earlier this year serving a genuine llms.txt from behind a bot-wall that returned 404s to programmatic clients. Observed traffic is how that gets caught, and if your visibility work and your security configuration have never been discussed in the same meeting, the coverage panel is a cheap way to find out whether one is quietly undoing the other.
A fourth use is more forward-looking. The agent connections class counts agents reaching your service directly rather than reading pages, and for most sites that number is currently small. It is worth watching, because it is the earliest signal available that agentic buying is arriving in your category, and it will move before anything else does.
Underneath all of it is the thing we have always been trying to build. GenSight exists to make AI visibility actionable rather than merely observable: to tell a brand what to fix, in what order, with the evidence behind it. Verification was the piece that was missing, and with observed traffic the loop closes, from diagnosis to action to whether the engines came back. Being read is the first step. From there it is being understood, then cited, then recommended, and that is the path the roadmap is built around.
AI Traffic is available in beta to all GenSight users. Connect a site and the first thirty days of observed crawler activity are available immediately, with the coverage panel usually being the most immediately actionable view.