Bot log analysis

AI crawler traffic is now a first-order phenomenon: Vercel and MERJ's study measured GPTBot making 569 million requests in a month across Vercel's network alone, and ClaudeBot 370 million, around a fifth of Googlebot's volume between them and climbing. Your share of that attention is sitting in your access logs, unread. The findings are usually actionable within an hour: a bot-protection rule challenging every AI fetcher, a training crawler consuming bandwidth you meant to refuse, or the quieter discovery that OAI-SearchBot has never visited at all, which single-handedly explains an absence from ChatGPT's citations. User-action agents like ChatGPT-User and Perplexity-User deserve special attention, because each hit is a live conversation about you. Vendors publish IP lists for their bots, so genuine traffic can be verified against spoofers, and a simple monthly review beats any amount of speculation about what the engines can see.