AI crawler readiness

Known crawler matrix

Sitemaps in the paste

• https://example.com/sitemap.xml

Two records that belong in different columns

After AI assistants start showing up as a referral source, a site has two facts to store apart from each other. The first is the path rule in robots.txt for a specific user-agent. GPTBot, ChatGPT-User, ClaudeBot, PerplexityBot and Google-Extended are names on this page’s known list, and each one can hit its own group. The second is referral visits that already happened. Vaneform’s AI referral board ranks sites by estimated visits from generative assistants and keeps the month and the estimate scope with the row. An allow verdict records what the pasted file says about the current path. A board figure records an estimate for its own period. When both sit in one note, give them separate columns: user-agent and matched rule in one, referral visits, month and confidence in the other.

GEO work often starts from the file, because an assistant can only cite a URL it was able to fetch, and a training crawl can only keep a URL the file allowed. That still leaves the referral question to the traffic lookup. Open the domain on the homepage checker for the whole-site estimate, then open the AI referral board for the assistant channel. The guide on measuring AI referral traffic walks through how to line those visits up with a content task. This page stops at the file.

GPTBot and Googlebot can take different groups

This checker uses the same matcher as the robots.txt tester. A group whose user-agent equals the name, ignoring letter case, is used first. Otherwise the * group is used. With neither group, the path is allowed. The longest matching Allow or Disallow wins. An empty Disallow is skipped, so a group whose only Disallow line is blank does not block the path. A file can therefore allow / in the * group or in a Googlebot group and still disallow / for GPTBot alone. The sample in the box is arranged that way. Change the path to /private and the longer Disallow: /private inside the * group outranks the shorter Allow: /. The Googlebot group in the sample only has Allow: /, so /private stays allowed for Googlebot: that group does not fall through to * once it exists.

Googlebot-Image is a separate name on the list. A group written for Googlebot does not apply to Googlebot-Image. With no Googlebot-Image group, it uses *. The server log notes treat Google-Extended as a robots.txt token and record Gemini fetches under Googlebot. A Disallow: / on Google-Extended changes that token’s rule in the file. Whether the homepage is open to Googlebot is the Googlebot group, or * when that group is absent. Bingbot is on the search list here; the same log notes record it as the Bing index user-agent, also used by Copilot.

What the matrix is counting

The matrix covers the tester’s known list and nothing beyond it: Googlebot, Googlebot-Image, Bingbot, GPTBot, ChatGPT-User, ClaudeBot, anthropic-ai, PerplexityBot, Google-Extended, Bytespider and CCBot. Googlebot, Bingbot, GPTBot, ChatGPT-User, ClaudeBot and PerplexityBot also appear in the site’s AI traffic log classifier. anthropic-ai, Bytespider and CCBot appear on the site-profile AI crawler list and in this tester. Those lists explain where the row names come from. The allow or deny on a row comes from the pasted text and the current path. The page does not request that domain’s robots.txt, and it does not record that a crawler has visited.

A blank paste has no groups, so every known name is default allow. Parse notes list local syntax issues such as a missing colon or an Allow before any User-agent. Sitemap lines are copied out of the paste. A user-agent you type on the robots.txt tester can sit outside this list; use that page when the name you care about is absent here. OAI-SearchBot, Claude-SearchBot and Perplexity-User appear in the traffic log classifier and are absent from this known list, so this matrix has no row for them until the shared list includes them.

Walk the sample on / and on /private

The text already in the box is a sample, written for this page so the chart has something to count. Replace it with a robots.txt you copied when you want a file from a live host. On path /, GPTBot uses its own group and matches Disallow: /, so the row is disallowed. Google-Extended does the same with its own Disallow: /. Googlebot uses its own group and matches Allow: /, so the row is allowed. ChatGPT-User is allowed by its own Allow: /. ClaudeBot’s group contains both Allow: / and Disallow: /private; only Allow: / matches /, so ClaudeBot is allowed. Every other name on the list has no group of its own and falls through to *. In that group Allow: / matches and Disallow: /private does not, so those rows are allowed. The chart for / therefore shows one disallowed name among the logged AI crawlers (GPTBot) and one disallowed robots.txt token (Google-Extended).

Press Test /private and the same file changes the * group. Both Allow: / and Disallow: /private match /private, and /private is the longer pattern, so the * rows become disallowed. ClaudeBot’s own group contains that same longer disallow, so ClaudeBot becomes disallowed too. Googlebot and ChatGPT-User still have only Allow: / in their groups, and a named group does not fall through to *, so both stay allowed. GPTBot and Google-Extended stay disallowed because Disallow: / matches every path. That is the whole analysis the chart can do: retotal the eleven known names after the path changes. Copy a real robots.txt over the sample when you want the same table for a file you maintain.

The llms.txt box reads the paste

The second panel counts three things in whatever you paste: a Markdown heading, an http or https link, and the characters /llms.txt. It also shows the character count and the line count. Vaneform publishes its own agent navigation at /llms.txt, and indexable pages point at it with rel="describedby". When a client lists text/markdown at the highest q, report pages, the methodology page and the guides respond with Content-Type: text/markdown; charset=utf-8, and the HTML head also carries a text/markdown alternate. That negotiation is already described in the public llms.txt file. It describes Vaneform’s own pages.

A paste from another host proves only that those characters are in the browser now. A heading and a link are present or absent as text. They do not establish that the host serves the file at /llms.txt, which status it returned, or which crawler fetched it. When a site-profile probe has stored a domain’s /llms.txt, that observation keeps the HTTP status and a preview on the public-file record. This page leaves that record closed. Use the paste to see whether a draft you are about to publish contains a title and a link; use a stored probe, or a fetch you run yourself, for the live URL.

What to write down, then where to look next

Keep the domain, the path, the user-agent, the group that was used (named, * or default), and the matched Allow or Disallow text. Add the date you copied the file. A later edit to robots.txt changes the verdict while an older traffic month stays that month, so the note needs both dates. The chart adds those rows: search crawlers, the AI crawlers named in the traffic log, the Google-Extended token, and the remaining names on the site-profile list. The list buttons under the chart filter the table to the same rows.

The next lookup is the domain on the homepage checker, then the AI referral board, then the guide that measures AI referral traffic. The homepage returns the visit estimate with its sources and dates. The board returns the assistant-channel estimate and the growth order. The guide is the place to decide what a change in that channel means for a page you publish. Three questions stay separate: how this file rules the path, what the estimate says, and how you will record the referral.

Use the verdict with a traffic lookup

Look up a domain →AI referral board →Measure AI referral traffic →robots.txt tester →More utilities →