Features / AI and agent readiness
Every site you look after is checked for the conventions an AI agent looks for: whether it can be discovered, whether its content can be read, what it says about crawlers, and what it can actually do. Nothing here is graded as a fault — publishing none of it is an ordinary website, and for plenty of clients refusing the crawlers is the brief.
Which of our clients are ready to be read by something that is not a person?
If you look after thirty sites, answering that today means opening thirty
robots.txt files by hand, and then a second pass for the half-dozen
/.well-known/ paths that have appeared in the last two years. Nobody does it,
so nobody knows — and it is starting to be asked, by clients who have read that
ChatGPT is sending traffic and want to know whether they are in it.
This check answers it for the whole estate at once, and re-answers it every day.
This is the one area here that never grades. A site with no llms.txt, no MCP
card and no opinion about GPTBot is a completely ordinary website, not a broken
one. Reporting it in red would be inventing a problem in order to sell the fix.
More to the point, for a large share of client work blocking is the brief. A publisher, a photographer, a membership site — plenty of people have decided, correctly for them, that they do not want their content used as training data. A tool that scored that decision as a failure would be a tool arguing with its user.
So findings are recorded as present or absent, allowed or blocked, and the judgement stays with you.
Whether /llms.txt is served, and whether it is served as text rather than as
an HTML page pretending to be one. It is a curated reading list: an agent that
finds one can go straight to the six pages that hold the substance instead of
crawling four hundred.
Whether the site will answer Accept: text/markdown with Markdown, or whether
it hands back a page of framework markup regardless. A model reading a site
through a browser-shaped fetch pays for the CSS classes, the inline SVG and the
JavaScript payload in order to recover a few hundred words of prose. Answering
the request it actually made is the difference.
robots.txt is read for eighteen AI crawlers by name — GPTBot,
OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, Google-Extended,
PerplexityBot, CCBot, Bytespider, Applebot-Extended, meta-externalagent,
Amazonbot and the rest.
Naming any of them at all is the signal worth having. It means somebody made a decision, rather than inheriting whatever the CMS shipped with in 2019. Whether they were allowed or blocked is reported, never judged.
Content Signals are read alongside them — the newer convention for saying what a crawler may do with what it takes: search, train, or answer directly. Published or not published, it is worth knowing which.
The capability layer, which is where this stops being about content and starts being about software:
/.well-known/api-catalog — how a site tells an agent what it offers, instead
of leaving it to infer from the HTML./.well-known/mcp/server-card.json — an MCP server card, the way a site hands
an agent a set of tools rather than a set of pages./.well-known/agent-skills/index.json — a skills index, the way a site hands
over its own procedures instead of hoping they are guessed./.well-known/http-message-signatures-directory — whether the site is set up
to verify signed requests, which is how it will eventually tell a real agent
from something wearing its user agent.Almost nothing publishes these yet. That is exactly why the check is worth having now: the estate-wide answer today is a baseline, and the first client who asks will be asking against it.
An agency that can answer "here is where each of your sites stands on AI access,
and here is what changed this quarter" is having a different conversation from
one that cannot. The check is the same daily pass that reads the certificate and
the DMARC record, so the answer is already there when the question arrives — and
a change is written into the site's history with the date, so when a client's
robots.txt started blocking Google-Extended is a fact rather than a
recollection.
No, and anybody telling you a .well-known file will is selling something.
Publishing these conventions makes a site cheaper and more reliable to read for
an agent that already wants to read it. What gets a site cited is the same thing
that always did: being the clearest available answer to the question.
It is a convention with real adoption, not a ratified standard, and the same is true of most of what is checked here. That is stated plainly rather than papered over — this is a young area, and a monitoring tool that presents a proposal as a requirement is doing the same thing it criticises header scanners for.
That is a decision about your client's business, not a technical one, and this will not make it for you. What it will do is tell you what each site currently says, so the decision is at least deliberate.
The crawler this service runs identifies itself in its user agent and fetches
only the site's homepage, robots.txt and the handful of .well-known paths
listed above. It does not spider the site.
All of them, including the free one. Every check runs on every plan for every site; the paid plans differ by how many sites you watch and how often the uptime checks run, never by what is looked at.