Admin — agent access policy

Toggle what each bot class may see. robots.txt and live responses update immediately. The policy is persisted to /agent-policy.json.

Training crawlers

Bulk crawlers that collect content to train foundation models.

GPTBot ClaudeBot CCBot Google-Extended Applebot-Extended Meta-ExternalAgent Bytespider Amazonbot DeepSeekBot

Access level

AI search indexers

Crawlers that index content so AI search engines can cite and link it.

OAI-SearchBot PerplexityBot DuckAssistBot YouBot

Access level

User-triggered agents

A real person asked a question right now; the agent fetches on their behalf.

ChatGPT-User Claude-User Claude-Web Perplexity-User

Access level

Autonomous / other agents

Standalone agents and unidentified bots (includes ora-agent and generic crawlers).

ora-agent

Access level

Current defaults (prod): no class is blocked, so no agent receives a 403. Training crawlers get metadata (200, no bodies) and Content-Signal: ai-train=no. Flip training to blocked to demo the 403 path.