AI Crawler Access Checker: Can AI Bots Read This Page?

Whether an AI bot can read a page is decided one path at a time.

A site's robots.txt can be open at the root and closed on the page that matters, or the other way round. Here is how one URL gets its verdict, which bots ask, what they take away, and why a blocked page is sometimes the right answer.

Site access and page access are different questions

Most AI crawler tools answer a site-level question: does robots.txt shut a bot out of the whole domain? That is the right first check, and the AI Crawler Checker runs it; the Robots.txt Checker reads the file itself line by line. Neither tells you about a particular page. A file that allows everything at / can still disallow /resources/, /clients/ or a single campaign URL, and a bot that is welcome on the home page may be turned away from the article you most want quoted.

This checker answers the narrower question. You give it one URL; it splits off the path, reads the site's robots.txt, and judges that exact path for each agent. Then it fetches the page once, so you also see what a crawler would receive if it were allowed in. Per-URL checking is not new (TechnicalSEO and serp.tools offer it, and both also send live requests as each bot, which this tool does not). What this page adds is the rule that decided each verdict, and a reading of the result as a policy you can confirm or change.

How robots.txt matching works for one path

The file is a list of groups. Each group opens with one or more User-agent lines and carries Allow and Disallow patterns. RFC 9309, the standard written down in 2022, and Google's open-source parser decide a single URL in two steps.

First, the group. An agent looks for a group that names it. If one exists, the agent follows that group only and ignores every other line in the file, including the * group. If none names it, it follows *. If there is no * group either, or no file at all, nothing restricts it. This checker matches names without regard to case and takes the longest name that fits, so a group for GPTBot is not confused with one for ChatGPT-User.

Second, the rule. Inside that group every pattern is compared with the path, query string included. A pattern matches from the start of the path; * stands for any run of characters and a trailing $ anchors the end. Of the patterns that match, the longest one decides. When an Allow and a Disallow of the same length both match, Allow wins. An empty Disallow: matches nothing.

# path: /insights/fund-ii-update/
User-agent: *
Disallow: /insights/         # matches, 10 characters
Allow: /insights/fund-       # matches, 15 characters: this rule wins

User-agent: GPTBot
Disallow: /                  # GPTBot reads only this group: blocked

So in that file ClaudeBot and PerplexityBot may read the page, because the longer Allow beats the shorter Disallow in the * group, and GPTBot may not, because its own group replaces * entirely. The card on this page prints that reasoning for each agent: which group it took, and the line that settled it.

The AI crawlers this checker names

Fifteen agents are judged on every run: the 14 AI user-agent tokens Verand's site audit watches, and Googlebot so you can compare the AI verdicts with the one that governs Google Search. The descriptions below are how each operator documents its agent, read on 27 September 2026. Check the operator's page before you block one, because the roles are not interchangeable.

Training crawlers, search crawlers and user fetches

The table falls into three jobs, and the difference is what makes a per-page decision possible. Training crawlers (GPTBot, ClaudeBot and the Google-Extended control, plus the bulk crawlers CCBot, Bytespider, Amazonbot and Meta-ExternalAgent) collect text that may end up in a future model. Search crawlers (OAI-SearchBot, Claude-SearchBot, PerplexityBot) build the index an assistant searches when it answers a question and links its sources. User fetchers (ChatGPT-User, Claude-User, Perplexity-User) retrieve a specific page because a person asked about it.

Operators document these as independent. OpenAI says outright that a site can allow OAI-SearchBot to appear in search results while disallowing GPTBot. That means a firm can keep its pages out of training sets and still leave them open to the crawlers that decide which pages an assistant can search. It also means a block written in a hurry, after a news story about AI training, often lands on the wrong agent. Whether any assistant then cites a page depends on far more than access, and nothing in robots.txt or on this card predicts it.

Page-level policy is a decision, not an error

Many AI crawler checkers treat every block as a mistake and hand you the lines to undo it. For a business that publishes to be found, a site-wide block usually is a mistake. A block on one path often is not. A regulated firm may want its insights, guides and disclosures open to AI search, and its investor portal, offering documents, client pages and form confirmations closed to every crawler. A thin page that exists to say "thanks, check your inbox" has nothing to offer an answer engine, and a page meant only for accredited investors was never written for one.

That is why the card reads the result as a policy question: is this the access you intended for this page? Willowdale Equity's robots.txt shows the pattern in practice. It opens with Allow: / for every agent, then disallows its build folders, design mockups and newsletter thank-you pages. Checked here, its articles come back allowed for all 15 agents, and its /thank-you/ page comes back blocked for all 15 by Disallow: /thank-you/. Both answers are the ones the file was written to give.

Two limits apply to any page you close this way. The file is public, so a Disallow line tells everyone the path exists; anything confidential needs a login, not a robots rule. And a blocked page can still appear in Google as a bare address if other pages link to it, because Google was told not to read it, not told to forget it. To keep a page out of the index, it needs a noindex tag that a crawler is allowed to fetch; the Noindex Checker reads that tag. A page that is both disallowed and noindexed, like that thank-you page, is making a request crawlers cannot read.

Allowed in robots.txt is not the same as reachable

robots.txt is an instruction, not a gate. It says what an agent is asked to do; it does not decide what the server answers. Two things can separate an "allowed" verdict from a bot actually receiving the page.

The first is infrastructure in front of the site. A CDN's bot management, a web application firewall or a hosting provider's security rule can answer a named AI agent with a 403 or a challenge page while the robots.txt says welcome. Some CDNs now offer one-click AI-bot blocking that lives entirely outside the file. This checker now fetches the page once with each crawler's user agent, beside the plain-browser request, and shows the answer as "reaches the page" next to each robots.txt verdict. That catches a block keyed on the user agent, which is how Cloudflare's AI bot policy works. A block keyed on the crawler's network addresses, which some hosts use for verified bots, is invisible to any test that is not the crawler itself; your CDN's firewall log is the definitive answer there.

The second runs the other way. OpenAI and Perplexity both say their user-triggered fetchers may not follow robots.txt, because a person asked for the page. So a Disallow line does not reliably keep ChatGPT-User or Perplexity-User out either. If a page must not be fetched by anyone, only authentication does that.

What the model sees: the served HTML

Access is half the question; the other half is what an agent takes away once it is in. The second half of the card shows the page the way a crawler that does not run JavaScript receives it: the HTML as served, with scripts and styles removed, and the main content separated from menus, header, footer and sidebars by the same rule Verand's site audit uses on every page it crawls.

That view matters because the AI crawlers have not behaved like browsers. Vercel's analysis of crawler traffic, published in December 2024, found that the major AI crawlers, OpenAI's and Anthropic's among them, fetched JavaScript files but did not execute them. A page that fills in its text after load can look complete in your browser and arrive nearly empty. A low count is not always a rendering problem, though. Verand's own help centre home page, checked here, returns 22 words of main content: a one-line intro and the names of its sections. It is served whole; it is simply a hub whose articles carry the text. The test is the comparison: if the main-content count on the card is far below what you see on screen, the text is arriving by JavaScript, and moving it into the served HTML is the fix.

A few limits of this reading. Whole-page word and byte figures come from the first 100,000 characters of HTML, so on heavier pages the card withholds them rather than print a short number; the main-content figures read the full page and stand. The text excerpt is capped at 20,000 characters in transit. And the noindex line reads the robots meta tag only, not an X-Robots-Tag response header.

robots.txt and llms.txt do different jobs

robots.txt tells crawlers what they may request. llms.txt is a proposed file that lists a site's most useful pages for AI tools in plain Markdown. It grants or refuses nothing, and no major AI operator has confirmed that its systems read it. It is best treated as a curated index of the pages a firm has reviewed and wants to stand behind. The llms.txt Generator writes one from your own pages; this checker decides nothing from it.

Common page-level mistakes

• A folder rule that catches a page you wanted open. Disallow: /resources with no trailing slash also blocks /resources-guide/ and /resources.html, because patterns match from the start of the path.
• Adding a named group to allow one path. A group for GPTBot with Allow: /insights/ does let it read your insights, and also lifts every other * rule for it. Repeat the restrictions you still want inside its group.
• Blocking the training crawler and expecting search to change. A GPTBot block does not touch OAI-SearchBot, and a Google-Extended block does not touch Google Search, AI Overviews or AI Mode, which use Googlebot's crawl.
• Counting on robots.txt to keep a page private. The file is public and user-triggered fetchers may not follow it. Confidential pages need a login.
• Disallow plus noindex on the same page. The crawler told not to fetch the page never reads the noindex tag. Pick one: noindex to leave the index, Disallow to save crawl effort.
• Query strings forgotten. Rules match the path and the query string together, so Disallow: /*? blocks every URL that carries a parameter, tracking links included.
• Reading the home page and assuming the rest. The verdict for / says nothing about /clients/. Check the pages that carry your expertise, one at a time.