Robots.txt Tester: Check Whether a URL Is Blocked
Paste your robots.txt, give it a URL, and see exactly which rule decides the outcome. Nothing you paste leaves your browser.
How the tester decides
Most robots.txt checkers tell you allowed or blocked and stop there. That is the least useful part of the answer. When a URL is blocked and you did not expect it to be, what you need is the line that did it, and this tool reports that: the rule, its text, and the line number it sits on.
The rules below are Google's, standardised as RFC 9309. They are worth understanding because the parts people get wrong are not obvious.
1. Finding the group that applies
A robots.txt file is a list of groups. Each group starts with one or more User-agent lines and is followed by the rules for those agents. The rule that trips people up is where one group ends and the next begins:
User-agent: Googlebot
User-agent: Bingbot
Disallow: /admin
User-agent: *
Disallow: /privateThat is two groups, not three. Consecutive User-agent lines extend the same header, so Googlebot and Bingbot share one set of rules. The first Disallow closes the header, so the next User-agent starts a fresh group.
A crawler uses exactly one group: the most specific one naming it. Googlebot in the file above obeys Disallow: /admin and completely ignores Disallow: /private, because the catch-all group does not apply once a specific group exists. People regularly add a rule to the * group and wonder why Googlebot ignores it.
Matching an agent name is case-insensitive and works on prefixes, so a group for Googlebot also covers Googlebot-Image when nothing names the image crawler directly. If two groups could apply, the longer name wins. If several groups name the same agent, their rules are merged.
2. Matching the path
Only the path and query string are compared. The scheme, the host and any #fragment take no part.
Two characters are special. An asterisk stands for any run of characters, and a dollar sign at the very end anchors the match to the end of the URL. A dollar anywhere else is just a dollar sign. Everything else, including dots, is literal, which is why /a.b does not match /axb.
Patterns are prefixes unless you anchor them. Disallow: /fish blocks /fish, /fish.html, /fishheads and /fish/salmon.html — anything that starts that way. It does not block /catfish or /desert/fish, because the match is anchored at the start.
Paths are case-sensitive. Disallow: /Admin does nothing about /admin. Agent names are case-insensitive; paths are not. Getting those the wrong way round is a common and completely invisible error.
3. Deciding which rule wins
When more than one rule matches, Google does not read top to bottom. It picks the longest pattern, counting the wildcard and anchor characters toward the length. If an Allow and a Disallow are exactly the same length, Allow wins.
That is why this works:
User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.phpBoth rules match /wp-admin/admin-ajax.php. The Allow is longer, so the file stays reachable while the rest of the admin area stays blocked. Order is irrelevant — you can write those two lines either way round and the outcome is identical.
It also explains a pattern that looks like a contradiction and is not:
User-agent: *
Disallow: /
Allow: /$That blocks the whole site except the home page. /$ is two characters against one, so it wins for / exactly, and it matches nothing else because of the anchor.
4. Where crawlers genuinely disagree
Longest-match is Google's rule, not a universal one. Plenty of crawlers use first-match instead: the first rule in the file that matches decides, and everything after it is ignored. On a file where Allow and Disallow overlap, the two approaches reach different answers.
There is also a case Google's own documentation calls undefined. Compare Allow: /page with Disallow: /*.htm for the URL /page.htm: both match, and comparing the length of a pattern that expands to an unknown width is not meaningful. When the tool sees a contest like that, it says so rather than inventing a verdict. If you get that warning, rewrite the rules so they do not overlap.
5. The traps worth knowing
- An empty group allows everything.
User-agent: Googlebotwith no rules under it still wins group selection, so Googlebot then ignores every restriction in the catch-all group. Usually the opposite of what was intended. - An empty
Disallow:blocks nothing. It is the conventional way to say "allowed", not a mistake. - Blocking is not hiding. A disallowed URL can still appear in search results, listed without a description, if other pages link to it. See noindex vs disallow for what to use instead.
Noindex:in robots.txt does nothing. Google stopped honouring that unofficial directive on 1 September 2019. Files still carry it, and the pages it was meant to hide are indexable.- Wildcards do not work in user-agent names.
User-agent: Google*is matched literally and therefore matches nothing. - Google ignores
crawl-delay. Bing honours it. If you need Google to slow down, the lever is server response time, not this file. - Only the first 500 KiB is read. Anything past that is ignored.
- A blocked page cannot be read at all, which means any
noindextag inside it is never seen. This is the single most common way pages get stuck in the index.
6. What this tool cannot do
It cannot fetch your live robots.txt, because that would mean sending your domain to a server and the whole point is that nothing leaves your browser. It also could not reach a staging site behind a login, which is often what you most want to test.
It reports what the file you pasted says right now. Google caches robots.txt for up to 24 hours, so a file you fixed this morning may not be the one Google is acting on. To see the copy Google actually holds, use the robots.txt report in Search Console.
Need to write the file rather than check it? The robots.txt generator builds one from a few questions.
Frequently asked questions
Does this send my robots.txt anywhere?
No. The parser and the matcher are JavaScript running on your own machine. Nothing you paste is uploaded, logged or stored, which is also why the tool cannot fetch your live robots.txt for you.
Why does it ask me to paste the file instead of fetching it?
Fetching your site would mean sending your domain to a server. It would also fail for anything behind a login or not yet public, which is often exactly what you need to test. Pasting costs you one copy and keeps everything on your device.
Google says a URL is blocked but this says allowed. Who is right?
Check three things. That you pasted the live file rather than a local copy, that you used the same user-agent, and that the URL matches exactly including its query string and letter case. If they still disagree, the file has probably changed since Google last fetched it; Google caches robots.txt for up to 24 hours.
Which crawler rules does this follow?
Google's, which are the rules written down in RFC 9309: the longest matching pattern wins, and Allow wins an exact tie. Some crawlers instead obey the first rule that matches, top to bottom, and will disagree on files where Allow and Disallow overlap. The tool flags those cases rather than pretending there is one answer.
Is robots.txt a way to keep a page private?
No, and it never was. The file is public, so listing a path advertises it. Blocking a URL also does not remove it from search results; it only stops the page being read. Use a password for anything private and a noindex tag for anything you want out of search.