robots.txt Tester
NewPaste a robots.txt file and test whether a specific URL path is allowed or disallowed for a given crawler user-agent, following real precedence rules.
Frequently Asked Questions
How does the tool decide which rule "wins" when multiple rules match?
It follows the standard robots.txt precedence rule: the longest matching path pattern wins, regardless of whether it's an Allow or a Disallow directive — this matches how Google and most major crawlers actually interpret conflicting rules.
What if my robots.txt has a specific group for one crawler and a wildcard group for everyone else?
The tool first looks for a group whose User-agent line exactly matches the crawler you're testing (case-insensitively); if none exists, it falls back to the User-agent: * group, matching real crawler behavior.
Does it support the * wildcard and $ end-of-string anchor in paths?
Yes — Disallow/Allow path patterns are converted to the equivalent regular expression, where * matches any sequence of characters and a trailing $ anchors the match to the exact end of the URL path.
What does it mean if no rule matches at all?
Per the robots.txt specification, the default behavior when no Allow or Disallow rule matches a path is to allow crawling — the tool reports this explicitly rather than treating an unmatched path as a disallow.
Can I test the same robots.txt against multiple user-agents?
Yes — just change the User-agent field and the result re-evaluates instantly against the same pasted robots.txt content, so you can quickly compare how Googlebot, Bingbot, and a generic wildcard crawler are each treated.