AI Crawler robots.txt Builder

New

Build a robots.txt that allows or blocks GPTBot, ClaudeBot, PerplexityBot, Google-Extended and other AI crawlers separately for training and search.

ai crawler robots.txtblock gptbotallow claudebotgoogle-extendedai bots robots.txt

Shortcuts:⌘ KSearch?HelpD MDark modeG TAll tools

Input

Output

Continue with:

Frequently Asked Questions

Can I allow AI search but block training?

Yes. Vendors publish separate user agents for training, search indexing and user-triggered fetches, and the builder groups them that way.

Does robots.txt enforce anything?

No, it is a request that well-behaved crawlers honor. Use firewall rules for enforcement.

Is my data uploaded?

No. Everything runs locally in your browser.

Generate a valid SKILL.md with YAML frontmatter for Claude Code, Copilot, Codex and other Agent Skills compatible agents.

Validate a SKILL.md file against the Agent Skills specification: frontmatter, name format, description length and body size.

Build an Agent2Agent (A2A) Agent Card JSON for /.well-known/agent-card.json with skills, interfaces, capabilities and security.