A misconfigured robots.txt can silently block search engines from pages you want indexed — or worse, leak sensitive URLs by listing them in a Disallow rule. This tool lets you paste your robots.txt and test exactly how a specific crawler will treat a specific URL path, before you push it live.
What Is the robots.txt Tester?
Paste your robots.txt content, pick a user-agent (like Googlebot or *), and enter a URL path — the tool evaluates the real precedence rules crawlers use (longest matching pattern wins, with wildcard and end-anchor support) and reports whether the path is allowed or disallowed, plus which exact rule decided it.
How to Use It
- Open the tool at /tools/robots-txt-tester.
- Paste your robots.txt content (or edit the default example).
- Enter the URL path you want to test.
- Enter the crawler’s user-agent (e.g.
Googlebot,Bingbot, or*). - Read the Allowed/Disallowed result and the matched rule.
Key Features
- Real crawler precedence logic — the longest matching Allow/Disallow pattern wins, matching how Google and other major crawlers actually resolve conflicts.
- User-agent group matching, falling back to the wildcard
*group when no exact match exists. - Wildcard (
*) and end-anchor ($) pattern support in Disallow/Allow paths. - Default-allow behavior when no rule matches at all, per the robots.txt specification.
When to Use This Tool
- Before deploying a new robots.txt to confirm it doesn’t accidentally block important pages.
- Debugging why a page isn’t being crawled, by testing the exact path against the exact crawler in question.
- Comparing behavior across crawlers (Googlebot vs. a generic wildcard bot) when you have crawler-specific rules.
Try It Now
Open robots.txt Tester directly, or press ⌘K and type the tool name from anywhere on the site.
Part of the Generators collection on WebdevToolbox — free, browser-based developer tools with no login required.