Free Robots.txt Generator

Robots.txt examples, and what each line tells a crawler.

A robots.txt is short, public and read before anything else on your site. Here is where it goes, the five lines a generator writes, the presets and when each one is right, and the one precedence rule that quietly undoes most hand-edited files.

Where the file lives

One file, at one address: https://yourdomain.com/robots.txt. Crawlers only ever request it at the root of a host, so a copy in /blog/robots.txt is never read, and a rule in it does nothing. Google puts the scope precisely: the rules "apply only to the host, protocol, and port number where the robots.txt file is hosted." So shop.yourdomain.com needs its own file, and so does the http:// version of a site if it still answers on its own rather than redirecting.

Two practical limits. Google reads the first 500 KiB and ignores the rest, which no hand-written file comes near but a generated list of thousands of paths can. And Google "generally caches the contents of robots.txt file for up to 24 hours", so a change is not live for Google the moment you upload it. Plan edits around that, particularly the ones that open something up.

The five lines a generator writes

Robots.txt syntax is a list of field: value lines. Field names are not case-sensitive; paths are, so /Admin/ and /admin/ are different rules. Comments start with #. Everything this generator produces is built from five fields:

• User-agent opens a group and names the crawler it speaks to. * means every crawler that has no group of its own. Consecutive User-agent lines share one set of rules.
• Disallow is a path prefix the crawler should not request. Disallow: /cart covers /cart, /cart/ and /cartoons, because it is a prefix. An empty Disallow: blocks nothing.
• Allow carves an exception out of a Disallow. When two rules match a URL, the longer one wins, and on a tie Allow wins.
• Sitemap points crawlers at your XML sitemap. It belongs to no group and can sit anywhere, but it must be the full address: Google requires "a fully qualified URL, including the protocol and host".
• Crawl-delay asks a crawler to wait that many seconds between requests. It is not part of the standard, and support is uneven (more below).

Two wildcard characters work in paths: * matches any run of characters and $ anchors the end of the URL. They are defined in RFC 9309, the IETF standard the protocol finally got in 2022, and Google supports both. A robots.txt wildcard is the difference between blocking every PDF and blocking one folder:

# every URL ending in .pdf
Disallow: /*.pdf$

# any URL carrying a session parameter
Disallow: /*?sessionid=

# the folder, but not the one file inside it the page needs
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php

Robots.txt examples for the common cases

The presets in the builder are the four starting points almost every site needs. Each is a sensible base to edit, not a finished policy.

Allow all. Two lines, User-agent: * and Allow: /. It is what robots.txt allow all looks like when written down, and it behaves exactly like having no file: a crawler with no restriction may fetch everything. Writing it anyway gives you a place to put the Sitemap line and a record that the openness is deliberate.

User-agent: *
Allow: /

Sitemap: https://yourdomain.com/sitemap.xml

WordPress. WordPress serves a virtual robots.txt of its own when no real file exists: Disallow: /wp-admin/ with an exception for admin-ajax.php, which themes and plugins call from the public site. Used as a robots.txt generator for WordPress, the preset writes exactly those two rules. It does not block /wp-includes/ or /wp-content/, because both hold the CSS and JavaScript your pages load, and Google renders pages with them. If an SEO plugin already manages the file for you, start from your live file instead, so the plugin's lines are what you edit.

Block everything. Disallow: / under *, the staging preset. More on it next, because it is the most useful and the most dangerous line in the protocol.

My live file. Appears after "Start from my file" and restores exactly what your site served, so you can always return to it after trying a preset.

Robots.txt disallow all: when it's right, and what it doesn't do

User-agent: * followed by Disallow: / asks every crawler that honours the file to request nothing. It is right for a staging copy, a development host, or a site that is not ready to be seen. It is wrong everywhere else, and the usual way it ends up on a live site is a staging file copied to production along with everything else. It stops Googlebot too, which is why the builder flags it every time it's in the output.

What it does not do matters as much. It is not a lock: the file is public and compliance is voluntary, so a staging site that must stay private needs a password. And it does not remove pages from Google. A blocked URL can still be indexed if other pages link to it; Google shows the bare address, because it was told not to read the page. Keeping a page out of the index is the job of a noindex meta tag or X-Robots-Tag header, which the crawler has to be allowed to fetch before it can see. Blocking a page and asking for it to be de-indexed is the classic way to get neither, and Google stopped reading noindex written inside robots.txt in 2019.

Why a named group has to repeat your * rules

This is the rule most generated files get wrong, and the reason this generator writes groups the way it does. Under RFC 9309 a crawler obeys the group that names it and ignores User-agent: * entirely. It does not merge the two. So this file, which looks like "keep everyone out of the client portal, and make sure GPTBot is welcome", does something else:

User-agent: *
Disallow: /client-portal/

User-agent: GPTBot
Allow: /     # GPTBot never reads the * group, so /client-portal/ is open to it

Every tool that adds a per-crawler group from a dropdown can produce that file. This one copies each line of your * group into every Allow group it writes, Allow and Disallow alike, plus the Crawl-delay, and adds a comment to the file saying so, because the copy only holds while the two stay in step. Change the * rules later and the named groups need the same change. Block groups are the exception: they are Disallow: / alone, since an Allow line copied into a block would reopen exactly the path it names.

A crawler your live file already names is treated differently. A group for a specific crawler is the one explicit statement of intent a robots.txt can make: counsel's decision, a CDN setting, a previous agency's policy. Writing a second group for it would be merged with the first and could reverse it. So the generator keeps that group exactly as written, comments included, and shows the crawler as "In your file". The Replace button beside it is there when you do mean to change it. It's the same rule Verand's own fix for blocked AI crawlers follows on customer sites.

Robots.txt crawl-delay: who reads it

Crawl-delay is the line most generators offer and fewest crawlers use. Google's documentation lists the fields it supports and says plainly that "other fields such as crawl-delay aren't supported." Anthropic's says the opposite for its crawlers: "we support the non-standard Crawl-delay extension to robots.txt." So a delay can slow some crawlers and will never slow Googlebot. If Google is putting real load on your server, its documentation points to answering with a 503 or 429 status for a while, not to robots.txt. One more reason the generator copies the delay into each Allow group: a named crawler that honours Crawl-delay only reads it from its own group.

AI crawlers are one section of the file, not a separate file

The AI crawlers in the builder are the 14 names Verand's site audit watches on every customer site, read from the product's own list when this page is built, so the two can't drift apart. They do different jobs: some collect pages for model training, some build the index an assistant searches, and some fetch a page because a person just asked a question. That split decides what a block achieves. OpenAI says of its user-triggered fetcher that "robots.txt rules may not apply", and Perplexity says its own "generally ignores robots.txt rules". Google-Extended is a control token rather than a crawler, and Google says it "does not impact a site's inclusion in Google Search". The Block training crawlers button sets the five the list classes as training and leaves the rest alone; for the purpose-by-purpose policy, the Robots.txt Generator for AI goes crawler by crawler.

Lines never to write

• Disallow on the CSS or JavaScript a page needs. Google renders pages with their scripts and styles. Block /assets/ or *.js and Google reads a broken page. The builder warns on rules that look like it.
• A Disallow on the page you're trying to noindex. The crawler can then never fetch the page, so it never sees the noindex. Allow the crawl, serve the tag, and remove the page from your sitemap.
• A relative Sitemap line. Sitemap: /sitemap.xml is not a full address, and Google requires one. The builder flags any line that doesn't start with http.
• A path without its leading slash. Disallow: admin is not a path. Rules start with / or a wildcard.
• A rule above the first User-agent line. It belongs to no group and RFC 9309 says to ignore it. When you start from your file, lines like that are listed as not carried over rather than silently lost.
• A misspelled crawler name. Names match without regard to case, but GPT-Bot is not GPTBot. A group for a crawler that doesn't exist blocks nothing and looks like it does.

Start from the file you have

Most sites already serve a robots.txt, and most of those files contain a line somebody added for a reason: a search results path, a staging folder, an old campaign directory. A generator that starts from a blank form invites you to overwrite all of it. "Start from my file" reads what your site serves today and loads it: the * rules and Crawl-delay into the builder, the Sitemap lines into the sitemap box, and every group for a named crawler kept word for word. If the file declares no sitemap and one answers at a standard address, that address is filled in and the note says it was found, not declared. Comments outside those named groups, and any line the parser can't read, are not carried over, and the note under the input lists what was left out. If the file can't be read at all, because of a server error or a response that isn't a text file, nothing is imported and the note says so, rather than presenting part of a file as the whole.

Before you upload

Read the output once, top to bottom, with your own site in mind: the generator knows the rules of the format, not the reasons behind your paths. Save it as plain text named robots.txt, put it at the root of the host, and load https://yourdomain.com/robots.txt in a browser to see that it's served as text and not as an HTML page (a theme's 404 page returned as a 200 is read as no file at all). Then run the Robots.txt Checker against the live site. The file is a set of requests to crawlers that choose to honour them; it controls crawling, not rankings, and nothing about it guarantees what any engine does with your pages.