Free Robots.txt Generator for AI

Block AI training, keep AI answers: how to write robots.txt for AI crawlers.

A robots.txt for AI is not one line that says yes or no to "AI". It is a set of decisions about separate crawlers that do separate jobs, written so the rest of your file keeps working. Here is what each decision costs and where the file stops.

What robots.txt for AI actually controls

robots.txt is a plain text file at the root of a host that tells crawlers which paths they may fetch. It is organised in groups: a User-agent line naming a crawler, then the Allow and Disallow rules that apply to it. User-agent: * is the group for everyone not named elsewhere. AI crawlers read the same file as search engines, so a robots.txt for AI is the same file with groups for the AI crawlers you want to treat differently.

Two things follow from that. The file is a request, honoured by crawlers that choose to honour it, not a lock; anything that must stay private needs a login. And it works per host, so www.yourdomain.com, the bare domain and help.yourdomain.com each serve their own file. If you want to see what a host's current file already says to each AI crawler, the AI Crawler Checker gives a verdict per bot; this page writes the policy.

Three jobs: training, search and fetching for a user

Each major AI operator now runs more than one crawler, and they do different work. A training crawler collects pages that may be used to train future models. A search crawler builds the index an assistant searches when it composes an answer. A user fetcher requests one page in the moment, because a person asked a question that needs it. Blocking one leaves the others exactly where they were. These are the 14 names the builder covers, grouped the way their operators describe them:

The grouping is read from each operator's own documentation. Where an operator publishes nothing, as ByteDance does not for Bytespider, the crawler sits under Other and the note says so, rather than carrying a guess. The same list drives Verand's site audit, so a name added there appears here on the next build of this page.

Blocking AI crawlers without leaving AI answers

The most common request is to keep writing out of training sets while staying findable when a client asks an assistant a question. The preset "Allow AI search and user fetches, block training" writes exactly that: Disallow: / for GPTBot, ClaudeBot, Google-Extended, Amazonbot and Meta-ExternalAgent, and allow groups for OAI-SearchBot, Claude-SearchBot, PerplexityBot, ChatGPT-User, Claude-User and Perplexity-User. The Other crawlers (the legacy anthropic-ai token, CCBot and Bytespider) get no rule, so they follow your * group until you decide.

Why this split holds up: OpenAI says disallowing GPTBot "indicates a site's content should not be used in training generative AI foundation models", while sites that opt out of OAI-SearchBot "will not be shown in ChatGPT search answers". Perplexity says PerplexityBot "is not used to crawl content for AI foundation models". Anthropic describes ClaudeBot as collecting content that "could potentially contribute" to training, and Claude-SearchBot as improving "search result quality". Blocking the training names and allowing the search names is therefore two separate decisions, and each operator documents them separately. Other tools offer the same preset; the parts that decide whether your file still works are below.

The trap on the allow side: groups replace your general rules

Under RFC 9309, the robots.txt standard, a crawler obeys only the group that names it. If a group names GPTBot, GPTBot ignores User-agent: * entirely, including every Disallow in it. That is well known on the block side. It is rarely mentioned on the allow side, where it does the damage: adding a plain allow group opens every path your general rules closed.

# before
User-agent: *
Disallow: /wp-admin/
Disallow: /client-portal/

# a common "allow GPTBot" snippet, appended
User-agent: GPTBot
Allow: /

After that append, GPTBot may fetch /client-portal/ and /wp-admin/, because its own group says Allow: / and nothing else. The fix is to repeat the general rules inside the named group, which is what this builder does for every group it adds:

User-agent: GPTBot
Disallow: /wp-admin/
Disallow: /client-portal/

Willowdale Equity's live file shows why this matters on a real site. Its User-agent: * group carries an Allow: / and 13 Disallow lines, for build folders, design mockups and thank-you pages. A bare allow group for an AI crawler would open all 13 to it. Loaded here, every allow group repeats all 14 lines. The robots.txt fix inside Verand's product writes its groups the same way.

If your * group has a Crawl-delay, it is repeated too, since a named group loses that as well. And if your * group says Disallow: / for the whole site, the builder repeats that faithfully and warns you, because an allow group that repeats a site-wide block allows nothing, and the real problem is the line under *.

A crawler your file already names is left as written

When you start from your live file and it already has a group for, say, ClaudeBot, the builder does not add a second one. It shows that crawler as named in your file, with its current rule, and leaves it alone. The reason is that RFC 9309 merges two groups that name the same crawler, so a second group could quietly reverse the first, and a named group is usually a decision somebody made on purpose: counsel, a previous agency, or a CDN setting. To change it, edit that group in your file. The output says which crawlers were left and why, as a comment.

The lines robots.txt cannot enforce

A generator that lists every crawler as if a Disallow works on all of them is telling you something the operators do not. The notes beside each crawler in the builder come from the operators themselves:

• ChatGPT-User. OpenAI: "Because these actions are initiated by a user, robots.txt rules may not apply." A block for it is a statement of preference, not a control.
• Perplexity-User. Perplexity says this fetcher "generally ignores robots.txt rules", because a user requested the fetch. Claude-User is different: Anthropic says its bots honour robots.txt, and blocking Claude-User stops Claude retrieving your pages for its users.
• Google-Extended is not a crawler. It is a control token: Googlebot does the fetching, and the token says whether that content may be used for Gemini training and grounding in Gemini Apps and Vertex AI. Google says it "does not impact a site's inclusion in Google Search nor is it used as a ranking signal". Google's description of the token does not name AI Overviews or AI Mode, which are part of Search, so do not expect a Google-Extended block to keep your pages out of them.
• The network can override the file. A CDN, firewall or security plugin can refuse an AI crawler by name before robots.txt is ever read, so the file can say allow while the site says no.
• Blocking fetches is not blocking citations. A crawler you block cannot fetch your pages, but an engine can still cite your site from other sources, such as a search index or a link a user pastes.

A policy for a regulated firm

Generic presets assume a publisher. A firm that publishes expertise so prospects can find it has a different question: which machines can read our disclosures, and did someone block the assistants our prospects use to shortlist firms? A reasonable starting point by firm type, as a content decision and not legal advice:

• Financial advisers and syndicators. Allow search and user fetch, so the assistants that summarise your firm can also read the disclosures on the same pages. Decide on training separately; it has no bearing on whether you appear in answers.
• Law firms and accounting firms. The same split usually fits educational articles. If a client agreement or licence restricts reuse of certain content, keep that content behind a login rather than relying on the file.
• Medical practices. Patient information should never be on a public URL in the first place. For public health articles, allowing search and user fetch keeps them citable; blocking training is a legitimate choice for original writing.

Whatever you choose, write it down and re-check it after every site deploy, because robots.txt is the file that gets overwritten by a theme, a plugin or a staging copy.

After you upload the file

Changes are not instant. OpenAI says "it can take ~24 hours from a site's robots.txt update for our systems to adjust" for search results, and Perplexity says it "may take up to 24 hours" for its systems to reflect a change. Then confirm the file is live and says what you meant with the AI Crawler Checker, check one important URL with the AI Crawler Access Checker, and read the whole file line by line in the Robots.txt Checker. For the non-AI parts of the file, search engines and sitemaps, the Robots.txt Generator covers them.

Mistakes worth avoiding

• Blocking the user fetchers to stop training. It leaves every training crawler where it was, and it removes you from Claude's live answers without doing the same for ChatGPT or Perplexity.
• A misspelled or retired name. A group for a name no crawler sends does nothing. anthropic-ai survives mainly in older files.
• A second group for a crawler already named. The two are merged, and the result may not be what either author meant.
• Disallow: / under * to stop AI. It stops Googlebot too, which takes the site out of Google Search.
• Pasting over the whole file. Replacing a live robots.txt with a fresh one drops its Sitemap lines and every rule you did not retype. Start from your file instead, and the builder appends.