AEOscoreFree AEO score

Glossary / AI engines and crawlers

ClaudeBot

What is ClaudeBot?

ClaudeBot is Anthropic's training crawler: it collects public web content that may contribute to training Claude models. Anthropic documents that disallowing ClaudeBot in robots.txt excludes your future content from training datasets. It is one of three Anthropic bots, alongside Claude-SearchBot for search and Claude-User for user-triggered fetches.

Last updated

In practice

Why it matters for AI visibility

The same training-versus-search distinction that applies to OpenAI applies here, and Anthropic is unusually explicit about it: three named bots, three separate robots.txt decisions. Blocking ClaudeBot is a training opt-out only; it does not remove you from Claude's live answers, but what ClaudeBot reads today is a vote on what future Claude models will know about you by default. Claude's user base skews professional and technical, so B2B businesses in particular should think before excluding themselves from its training diet. Anthropic supports the Crawl-delay directive if server load is the worry, which is a gentler tool than a block, and publishes its crawler IPs at claude.com/crawling/bots.json for log verification. As with every AI bot decision, make it deliberately: the default posture many security tools ship, block everything unfamiliar, was not designed with your AI visibility in mind.

Keep reading

Related terms

What's your AEO score?

Send your domain and a named expert will run it through all six engines, score the results and email your report within 1 to 2 working days. Free, no card, no signup.

  • ChatGPT
  • Claude
  • Gemini
  • Perplexity
  • Google AI Overviews
  • Google AI Mode