Which AI crawlers can read your site?

The bots we check

• OAI-SearchBotOpenAI

  ChatGPT search results

• ChatGPT-UserOpenAI

  fetches pages for a user's question; may not honour robots.txt

• GPTBotOpenAI

  training crawler; blocking it does not remove you from ChatGPT search

• Claude-SearchBotAnthropic

  Claude search results

• Claude-UserAnthropic

  fetches pages for a user's question; may not honour robots.txt

• ClaudeBotAnthropic

  training crawler; a training opt-out only

• PerplexityBotPerplexity

  Perplexity's index and answers

• Perplexity-UserPerplexity

  fetches pages for a user's question; generally ignores robots.txt

• GooglebotGoogle

  Search, AI Overviews and AI Mode read what Googlebot fetches

• Google-ExtendedGoogle

  Gemini training and grounding control; not AI Overviews or AI Mode

How it works: we fetch yourdomain/robots.txt once (https, then http, following redirects only to public hosts), parse the user-agent groups and allow/disallow rules, and apply the longest-match rule for the path "/" the way crawlers do. A bot with no group of its own follows the "*" group; no file at all means everything is allowed. Search crawlers decide whether an engine can cite you; training crawlers are a separate opt-out; user agents fetch on a person's behalf and may not honour robots.txt. Nothing is stored. This is quick tip 07 in tool form. The live access test then fetches your homepage as each crawler, because a firewall can turn bots away without robots.txt ever showing it: the Cloudflare guide and the site builder guide cover the fixes.