Robots.txt Checker

What a robots.txt file does, and how one line hides you from AI answers.

The file is forty years younger than the web's oldest habits and still the first thing every crawler reads. Here is what it controls, what it does not, and the mistakes that cost sites their place in ChatGPT, Perplexity and Gemini.

What the file is

A robots.txt is a plain-text file served at the root of a site, https://yourdomain.com/robots.txt, and nowhere else. It is the Robots Exclusion Protocol, drafted informally in 1994 and written down as an IETF standard, RFC 9309, in 2022. A crawler that respects the protocol fetches this file before it fetches anything else on the host and reads it as a set of instructions: which parts of the site that crawler may request, and which it should leave alone.

The file is built from groups. A group starts with one or more User-agent lines naming the crawler the group speaks to, and continues with Disallow and Allow lines listing path prefixes. A Sitemap line can sit anywhere and points crawlers at your XML sitemap. Comments begin with #. That is the whole grammar.

# every crawler not named below
User-agent: *
Disallow: /admin/
Disallow: /cart

# one crawler, its own rules
User-agent: GPTBot
Disallow: /

Sitemap: https://yourdomain.com/sitemap.xml

Two rules of the protocol decide almost everything. First, a crawler obeys the group that names it most specifically and ignores the rest: if there is a GPTBot group, GPTBot never reads the * group. Second, a crawler with no matching group at all has no restrictions. So a site with no robots.txt is fully open, and a robots.txt with an empty Disallow: is also fully open. Restriction only ever comes from an explicit line.

What it is for, and what it is not

The file exists to manage crawl load and to keep crawlers out of pages that are useless to them: search results pages, faceted filters, staging paths, cart and checkout flows, endless calendar archives. It was never a security mechanism. Every crawler that honours it does so voluntarily, and the file is public, so a Disallow: /private-report/ line is a signpost, not a lock. Anything that must stay private needs authentication.

It is also not the way to keep a page out of Google's index. A URL blocked in robots.txt can still be indexed if other pages link to it; Google simply shows the address with no snippet, because it was told not to read the page. The directive that removes a page from the index is a noindex meta tag or an X-Robots-Tag header, and both only work if the crawler is allowed to fetch the page and see them. Google stopped honouring noindex written inside robots.txt in 2019. Blocking a page and asking for it to be de-indexed at the same time is the most common way to get neither.

How one Disallow line blocks ChatGPT or Perplexity

AI assistants reach your site through named crawlers, and each operator publishes the name its crawler sends in the User-agent header. Those names are the whole point of this checker. A group like the GPTBot one above, two lines long, means OpenAI's training crawler requests nothing from your site. A Disallow: / under User-agent: * means every crawler that is not given its own group, which is usually all of them, stops at the door. On a site that publishes expertise so that people find it, that second case shuts out every AI crawler that honours the file, and the file does not warn you.

It happens quietly for three reasons. Blanket blocks are often installed by a hosting control panel, a security plugin or a developer protecting a staging copy, then copied to production. Named blocks are added when a story about AI training makes the rounds, without separating the crawlers that train models from the ones that fetch a page to answer a live question. And nothing in a normal analytics stack reports a crawler that never arrived. The site keeps ranking in Google, which uses a different crawler, so the block is invisible right up until someone asks an assistant a question you should have answered.

The 14 crawlers this checker names

These are the names Verand's site audit watches on every customer site, and the checker tests each one against your file. The role each plays is how its operator documents it; check the operator's page before you decide to block one, because the roles are not interchangeable.

The fix this page hands you lists the ten of those that belong to OpenAI, Anthropic, Perplexity and Google, whose assistants Verand tracks. The other four gather pages in bulk for datasets, model training and product indexes; they are reported so you know they are blocked, and whether to allow them is a policy choice about training data, and a fair one to make either way.

Google versus AI answers

The confusion that produces most bad robots.txt files is treating Google and the AI assistants as one audience. They are not. Googlebot is the crawler behind Google Search, and blocking it is the only way to fall out of Google. Google-Extended is a separate token that controls whether the pages Googlebot already fetched may be used for Gemini; Google documents that it has no effect on ranking or on inclusion in Search, and that AI Overviews and AI Mode are part of Search and draw on the same index. So a Google-Extended block does not remove you from Google's AI answers, and a GPTBot block does nothing to your Google rankings.

The practical consequence: a site can be perfectly visible in Google while being absent from ChatGPT and Perplexity, and the owner will see healthy Search Console numbers the whole time. The robots.txt is where that split is decided, one User-agent group at a time, which is why this checker reports by name rather than with a single pass or fail.

Common mistakes

• Disallow: / under * on a live site. The staging file that shipped. Everything is blocked, including Googlebot. The most damaging line in the protocol and the easiest to miss, because the file looks tidy.
• Relying on a blank line to separate groups. Under RFC 9309 a blank line doesn't end a group. Rules after it still belong to the User-agent lines above it; only a User-agent line that comes after rules starts a new group. Some older parsers did treat a blank line as a break, so keep each group's lines together and never rely on spacing to decide which crawler a rule applies to. Rules placed before any User-agent line belong to no group and do nothing.
• Expecting the * group to apply to a named crawler. Once GPTBot has its own group, the general rules no longer reach it. To keep the general restrictions and add one line for GPTBot, the whole set has to be repeated inside GPTBot's group.
• Serving an HTML page at /robots.txt. A theme's 404 page returned with status 200, or a login redirect. Crawlers treat it as no file, so every restriction silently disappears. This checker reports it as missing for the same reason.
• noindex in robots.txt. Unsupported by Google since 2019. Use the meta tag or the header, on a page the crawler is allowed to read.
• Blocking the user-triggered fetchers to stop training. ChatGPT-User, Claude-User and Perplexity-User arrive because a person asked a question you can answer, and none of them is a training crawler, so the block does nothing about training. It may not even hold: OpenAI and Perplexity say their user fetchers may not follow robots.txt. Claude-User does honour it, so there the block stops Claude fetching your pages for its users.
• Case and spelling. Names are matched case-insensitively, but GPT-Bot is not GPTBot. A misspelled group blocks nothing and gives false comfort.
• No Sitemap line. Not a block, but the cheapest signal you can add: one line pointing every crawler at the sitemap so discovery does not depend on your internal links.

What a good file looks like

For most sites that publish expertise: a * group that keeps crawlers out of the paths no reader should land on, a Sitemap line, and no site-wide Disallow for any crawler you want citing you. If you have decided to withhold training data from a particular operator, block its training crawler by name and leave its fetcher and search crawler open, so the decision is precise rather than a blanket outage. Then re-check after every deploy, because the file is the kind of thing that gets overwritten.