Robots.txt Generator

Build a robots.txt file one rule group at a time, block the AI training crawlers with one click, and check a path against the result before you deploy it. The tester uses the same longest-match rule as Google, so you see here that /admin/help/ is still crawlable, instead of finding out in Search Console weeks later.

Rule groups

Paths are matched from the start of the URL path, case-sensitively. * matches any run of characters, a trailing $ anchors the end. Crawl-delay is ignored by Google; Bing and Yandex read it.

robots.txt
Test a URL path against this file

1 Robots.txt Generator example

The Example button produces a default group for every crawler, one group that blocks the AI training bots, and a sitemap line. Groups with identical rules are merged under several User-agent lines, which RFC 9309 allows.

robots.txt

User-agent: *
Allow: /admin/help/
Disallow: /admin/
Disallow: /cart
Disallow: /*?sort=

User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: CCBot
User-agent: Applebot-Extended
Disallow: /

Sitemap: https://example.com/sitemap.xml

2 A crawler picks one group, then the longest rule

First the crawler picks the group whose User-agent matches its own product token, case-insensitively. It falls back to * only if there is none. A named group replaces the * group; it does not add to it. For example, once you add a Googlebot group, the rules under * no longer apply to Googlebot, so repeat any you still need.

Inside the group, the order of lines does not matter. The crawler compares every Allow and Disallow that matches the path, and the longest pattern wins. On a tie, Allow wins. If nothing matches, the path is allowed.

Paths tested against the example (plus Disallow: /*.pdf$)

/admin/help/faq      Allow: /admin/help/ (12 chars) beats Disallow: /admin/ (7)
/admin/users         Disallow: /admin/
/products?sort=price Disallow: /*?sort=
/products?page=2     no rule matches, allowed
/file.pdf            Disallow: /*.pdf$ matches
/file.pdf?v=2        /*.pdf$ does not match: $ means "ends here"

3 Robots.txt controls crawling and nothing else

It does not hide pages

Use noindex to keep a page out of results. A disallowed URL can still be indexed, without a snippet, if other sites link to it. Allow crawling and serve noindex, or put the page behind a login.

It is not access control

The file is public and compliance is voluntary. Listing /secret-admin/ in it points anyone curious straight at that path. Well-behaved bots obey the file, and scrapers can read it for targets.

Some tokens are not crawlers

Google-Extended and Applebot-Extended never fetch anything. Googlebot and Applebot read them to decide whether your content may be used for AI training. Blocking them does not affect search.

It covers one host

The file at https://example.com/robots.txt does not cover www.example.com, http://, or another port. Each origin needs its own file.

How to use the Robots.txt Generator tool

  1. Pick the Default: Allow all writes an empty Disallow for every crawler, and Disallow all writes Disallow: /.
  2. Edit the * group or click Add group for a specific bot. Choose it from the User-agent list or pick Custom and type the token. Put one path per line in Disallow and Allow.
  3. Click Block AI training crawlers to add Disallow: / groups for GPTBot, ClaudeBot, Google-Extended, CCBot and Applebot-Extended. Groups with identical rules are merged in the output.
  4. Add your sitemap URLs at the bottom, then type a user agent and a path into the tester to see whether that URL is allowed and which rule decided it.
  5. Click Copy or Download and upload the file as /robots.txt at the root of each host.

How to check a URL against robots.txt in code

JavaScript (Node 18+) 

// Longest match wins, Allow wins ties (RFC 9309, what Google does)
function toRegex(pattern) {
    const anchored = pattern.endsWith("$");
    const body = (anchored ? pattern.slice(0, -1) : pattern)
        .split("*")
        .map((s) => s.replace(/[.+?^${}()|[\]\\]/g, "\\$&"))
        .join(".*");
    return new RegExp("^" + body + (anchored ? "$" : ""));
}

function isAllowed(rules, path) {
    let best = null;
    for (const r of rules) {
        if (!r.path || !toRegex(r.path).test(path)) continue;
        if (!best || r.path.length > best.path.length ||
            (r.path.length === best.path.length && r.allow)) best = r;
    }
    return best ? best.allow : true;
}

const rules = [
    { allow: false, path: "/admin/" },
    { allow: true, path: "/admin/help/" },
];
console.log(isAllowed(rules, "/admin/help/faq")); // true
console.log(isAllowed(rules, "/admin/users"));    // false

This covers only the matching step. A real crawler first picks the one group whose User-agent matches its token, and falls back to * only when none does.

Python 

from urllib.robotparser import RobotFileParser

rp = RobotFileParser()
rp.set_url("https://example.com/robots.txt")
rp.read()

print(rp.can_fetch("Googlebot", "https://example.com/admin/users"))
print(rp.crawl_delay("Bingbot"))   # None when not set
print(rp.site_maps())              # list of Sitemap: URLs, or None

# Caution: robotparser applies rules in file order (first match wins)
# and does not understand * or $ inside paths, so results can differ
# from Google's for files that rely on either.

For Google-accurate results in Python, use a parser that implements RFC 9309, such as Google's own C++ robotstxt library or a port of it.

Checking the live file 

# Is it there, and is it plain text?
curl -sI https://example.com/robots.txt | grep -iE "^(HTTP|content-type)"

# Read it as a crawler would
curl -s -A "Googlebot" https://example.com/robots.txt

# A 5xx here makes Google stop crawling the site for a while;
# a 404 means "no rules", so everything is allowed.

Common errors and how to fix them

Rules under User-agent: * are not applied to Googlebot
Once a group names Googlebot, Googlebot reads only that group and ignores *. Copy the shared rules into the Googlebot group too, or drop the specific group.
Disallow: /private still lets /Private/ through
Paths are case-sensitive. Add both spellings, or fix the URLs so there is only one.
Disallow: /*.pdf$ does not block /file.pdf?download=1
The $ anchors the end of the whole path including the query string, so any query breaks the match. Use /*.pdf without the $ to block every URL containing .pdf.
A blocked page still shows up in Google with no description
robots.txt controls crawling, and Google can index a URL it has never fetched. It found links to the URL and indexed it without reading it. Remove the Disallow so Google can see a noindex tag on the page.
The whole site dropped out after a deploy
A staging robots.txt with Disallow: / shipped to production. Keep robots.txt out of the build that differs per environment, or generate it from an environment variable, and check it after each release.

Good to know

RFC 9309 (2022) turned the 1994 convention into a standard. It fixes the details that used to differ between crawlers: records are grouped by consecutive User-agent lines, matching of the user agent is case-insensitive while matching of paths is case-sensitive, the most specific (longest) rule wins, and crawlers must parse at least 500 KiB of the file. Google stops reading after 500 KiB, so anything below that line is ignored.

Crawl-delay, Host and Noindex are not part of the standard, and Google ignores all three. Sitemap lines are also outside the RFC, but every major search engine reads them, and they apply to the whole file regardless of which group they appear in.

Specifications and references

This tool follows the specifications below. Read them when you need the exact normative wording, for example to settle an edge case between two implementations.

Frequently Asked Questions

How do I block AI crawlers with robots.txt?

Add a group with Disallow: / for each bot's token: GPTBot (OpenAI), ClaudeBot (Anthropic), Google-Extended (Gemini training), CCBot (Common Crawl) and Applebot-Extended (Apple). The Block AI training crawlers button adds all five. Compliance is voluntary, so this stops only the companies that honor robots.txt.

Does blocking Google-Extended remove my site from Google Search?

No. Google-Extended is a token that Googlebot checks before your content is used to train Gemini models; nothing crawls under that name. Search crawling and ranking are unaffected. The same goes for Applebot-Extended and Apple's search features.

Does Google respect Crawl-delay?

No. Googlebot ignores the Crawl-delay line and adjusts its rate from how fast your server responds. Bing and Yandex do read it, so set it when one of those bots is loading your server too hard.

When two rules match, which one wins?

The longest matching path wins, regardless of the order of the lines. If an Allow and a Disallow are the same length, Allow wins. That is the rule in RFC 9309 and what Google implements, and it is what the path tester on this page uses.

Will Disallow keep a page out of Google?

Not reliably. A disallowed URL can still be indexed without a snippet if other pages link to it. To keep it out of results, allow crawling and add a noindex meta tag or an X-Robots-Tag header, because Google has to fetch the page to see either.

Related

Build the file your Sitemap line points at with the XML sitemap generator. To keep a page out of search results instead of out of the crawl, the meta tag generator adds noindex. To move URLs instead of blocking them, use the redirect generator.

More Generators

XML Sitemap Generator Turn a list of URLs into a valid sitemap.xml. Meta Tag & Open Graph Generator Generate SEO and social meta tags with a preview. Redirect Rule Generator Generate .htaccess and nginx redirect rules. URL Parser Split a URL into protocol, host, port, path, query and fragment. UUID / ULID Generator Generate UUID v1, v3, v4, v5 and ULIDs, up to 1,000 at a time.

Browse all generators

Copied to clipboard!