robots.txt

Your robots.txt now encodes commercial decisions most businesses never consciously made. The vendors themselves frame it this way: OpenAI and Anthropic document distinct tokens for training (GPTBot, ClaudeBot) versus search visibility (OAI-SearchBot, Claude-SearchBot), and Perplexity documents that its user-action agent generally ignores the file altogether, because a human asked. So the same file can, in four lines, refuse to feed future models while remaining fully citable in live answers, or, in one careless wildcard, delete you from AI search without anyone noticing for months. Audit it with three questions: which AI bots are named, does each rule reflect a choice someone actually made, and has a blanket block from the 2023 panic outlived its rationale? Remember its limits, too: robots.txt is a published request, not an access control, and it will not stop user-action fetches or bad actors. For those, the firewall is the tool; for policy, this file is.