Robots.txt Tester

A robots.txt file is a list of groups. Each group opens with one or more User-agent lines and continues with the Allow and Disallow rules that apply to those crawlers. A crawler reading the file does not combine every group it finds. It picks one set of rules and ignores the rest.

The tester follows the same selection order RFC 9309 describes:

1. The crawler's own token. If any group names the product token, for example GPTBot, that group applies. The comparison is case-insensitive, so gptbot and GPTBot are the same agent.
2. A parent token. If a hyphenated token has no group of its own, the tester falls back to the longest parent that does. Googlebot-Image uses the Googlebot group when there is no Googlebot-Image group, which mirrors how Google treats its sub-crawlers.
3. The * group. Only when neither of the above exists does the wildcard group apply.
4. Nothing. With no matching group and no * group, every URL is allowed.

The summary card names which of these happened, so you can see at a glance whether your crawler hit its own rules or fell through to *.

The point that trips people up most is that a specific group replaces the * group instead of adding to it. Say your file has Disallow: /private/ under User-agent: *, followed by a separate User-agent: GPTBot group containing only Allow: /. GPTBot may crawl /private/, because the rules under * simply do not exist for it. If a rule should apply to a named crawler, repeat it inside that crawler's group.

Two smaller details also come from the spec. Several groups naming the same agent are merged into one, so Googlebot rules split across the file behave as if they were written together. And consecutive User-agent lines before any rule share one group, which is how you give GPTBot and ClaudeBot identical rules without writing them twice.

The crawler field accepts a bare product token or a full User-Agent string. With a full string the tester tries to extract the token first. If the Group applied row shows the * group when you expected a named one, enter the bare token instead; that always works.

Once the group is chosen, the tester compares each URL path against every Allow and Disallow rule in it. The order of the lines does not matter. Length does: among all rules whose pattern matches the path, the one with the longest pattern decides. If an Allow and a Disallow of equal length both match, Allow wins. If no rule matches at all, the URL is allowed.

Take a group with four rules: Disallow: /shop/, Allow: /shop/sale/, Disallow: /*.pdf$ and Disallow: /*?sort=. Here is how some paths come out:

* matches any run of characters, including none and including slashes. A $ at the end of a pattern means the path must end exactly there, which is why /*.pdf$ blocks the PDF but not the same PDF with a query string attached. Without $, every rule is a prefix match: Disallow: /admin also blocks /administrator and /admin-login. Write /admin/ if you only mean the directory.

A Disallow: line with nothing after it blocks nothing. It is the traditional way of saying everything is allowed, and the tester skips it when matching. Disallow: / is the opposite case, the shortest pattern that matches every path, so any more specific Allow beats it.

For every URL the results table shows the winning rule with its line number, for example "Line 4: Allow /shop/sale/". When it says "No rule matched (allowed by default)", nothing in the applied group covers that path. Ties are rare, but they do happen with pairs like Allow: /page and Disallow: /page, and the tester resolves them toward Allow, the least restrictive choice the spec asks for.

AI companies publish their crawler tokens so site owners can address them in robots.txt: GPTBot for OpenAI, ClaudeBot for Anthropic, PerplexityBot, CCBot and others, plus Google-Extended, a control token Google reads to decide whether content fetched by its regular crawlers may be used for its AI models. Testing any of them works exactly like testing Googlebot. Type the token in the crawler field, paste the file and list the URLs you care about.

Group selection matters a lot here. Many sites start with a broad User-agent: * group and then add a short block for each AI crawler. Because a named group replaces *, an AI crawler with its own group ignores every rule written for everyone else, including the Disallow lines protecting admin or checkout paths. Running the same URL list for GPTBot, then ClaudeBot, then a crawler you never mentioned is a fast way to confirm each one sees what you intended.

Some files now also carry a Content-Signal line, part of the Content Signals Policy proposed by Cloudflare. It states how content may be used after it has been fetched, with three keys:

• search: building a search index and showing links and short excerpts in results
• ai-input: feeding content into AI answers at query time, such as retrieval or grounding
• ai-train: training or fine-tuning AI models

Each key takes yes or no, for example search=yes, ai-input=yes, ai-train=no.

The tester reports the Content-Signal value from the group applied to your crawler in the summary card, and it never changes a verdict. A signal is a preference; Allow and Disallow decide whether fetching is permitted. A crawler can be allowed to fetch a page whose signal says ai-train=no, and whether that preference is respected is up to the crawler's operator. Since the line lives inside a group, a crawler with its own group will not see a signal placed under * unless you repeat it there.

If you are writing these rules from scratch, an AI crawler robots.txt generator can produce the per-crawler groups, and pasting its output here lets you test them before anything goes live.

The parser flags lines that crawlers will ignore or misread and lists them under Parser warnings with their line numbers. These come up most often:

• Rules before any User-agent line. An Allow or Disallow at the top of the file belongs to no group, so crawlers drop it. A Content-Signal line placed above the first group is ignored for the same reason.
• Paths that do not start with / or *. Disallow: private/ is not a valid pattern. Write /private/.
• Crawl-delay. It is not part of RFC 9309 and Google ignores it. Bing does read it, so the tester shows a warning rather than treating it as broken.
• Vendor directives. Host and Clean-param are Yandex extensions. Noindex, Nofollow, Request-rate and Visit-time are flagged the same way. Google stopped honouring Noindex in robots.txt on September 1, 2019, so it does nothing there.
• Typos and unknown fields. Dissalow: /tmp/ is reported as an unknown directive. Crawlers skip it silently, which leaves /tmp/ open.
• Broken lines, such as a line with no colon or a User-agent: with no value.

Two mistakes produce no warning at all, because the syntax is valid.

First, paths are case-sensitive. Disallow: /Private/ does not block /private/, and the tester will show that path as allowed. Directive names are not case-sensitive, so disallow: and Disallow: behave the same, but the paths after them are compared character by character.

Second, blocking is not the same as removing from search. robots.txt controls crawling. A blocked page that other sites link to can still show up in results, usually without a description, because the crawler was never allowed to read it. To keep a page out of the index, it has to stay crawlable and carry a noindex meta tag or HTTP header.

One more trap comes from prefix matching. Disallow: /blog also blocks /blog-archive and /blogroll. When you test the path you mean to block, add a few neighbouring paths to the list as well and check that only the intended ones come back Blocked.

The tester evaluates the text you paste. It never fetches a live robots.txt, so a draft that has never been deployed, such as the output of an AI crawler robots.txt generator, can be checked the same way as a production file. It also ignores the domain of the URLs you list. A full URL is reduced to its path and query string, and the rules in your pasted file are applied to that whatever host the URL names. To test a live site, open its /robots.txt in a browser tab and copy the contents in.

It does not check whether a page is indexed, ranked or even reachable. Allowed means the rules permit fetching, nothing more. HTTP behaviour is outside its scope too. A real crawler that gets a 404 for robots.txt treats everything as allowed, and repeated server errors can make it treat the whole site as off limits for a while. Pasted text has no status code, so none of that is simulated.

Path normalisation is limited. Patterns are compared exactly as written, and the tester does not decode or re-encode percent escapes in your rules. When you paste a full URL, the browser's URL parser may encode characters such as spaces or accented letters, so a rule written with the raw character can miss the encoded path. For URLs with non-ASCII characters, write the rule in percent-encoded form, which is how crawlers request them.

The matching follows RFC 9309, including the * and $ special characters, and Google, Bing and the main AI crawlers share that core logic. Individual crawlers can still differ in edge cases, and some ignore robots.txt entirely. robots.txt cannot prove who a crawler is either, since any client can send a GPTBot user agent.

Everything runs in your browser. Nothing is uploaded, there is no account to create and the tool is free, so internal or unreleased files stay on your device. The per-URL results can be exported as CSV or as an image for a ticket or an audit report.