AEOscoreFree AEO score

Glossary / AI engines and crawlers

AI crawler

What is AI crawler?

An AI crawler is a bot an AI company sends to fetch web pages. They come in three kinds, and the difference matters: training crawlers collect content for building models, search crawlers build the indexes that answers cite, and user-action fetchers grab a page live because someone just asked about it. Each kind is controlled separately in robots.txt.

Last updated

In practice

Why it matters for AI visibility

Treating AI crawlers as one thing leads to expensive mistakes. Blocking everything with the word bot in it can pull your business out of ChatGPT and Perplexity answers entirely, while blocking nothing means donating your content to model training you may not want to support. The sane posture for most businesses seeking customers is the opposite of reflex blocking: welcome the search and user-action bots, decide deliberately about the training ones. There is also a hard technical fact to plan around: Vercel and MERJ's large-scale study found that no major AI crawler executes JavaScript, so anything your site renders client-side simply does not exist for them. Knowing which bots visit you, and what they can read, underpins every other AEO effort.

Measurement

How AEOscore measures this

AEOscore's scans check that your site is readable by the crawlers each engine relies on, and flag blocks that are costing you answer visibility.

Keep reading

Related terms

Quick tips that use this

Go deeper on AEOscore

Sources

What's your AEO score?

Send your domain and a named expert will run it through all six engines, score the results and email your report within 1 to 2 working days. Free, no card, no signup.

Get your free AEO scoreSee pricing
  • ChatGPT
  • Claude
  • Gemini
  • Perplexity
  • Google AI Overviews
  • Google AI Mode