Quick tips · Quick tip 07
Check your robots.txt for the AI bots.
This is the 20-second version. robots.txt says allowed but AI still can't read you? Check whether Cloudflare is blocking the crawlers
Blocked AI without knowing?
Check your robots.txt for the AI bots. If the bot can’t read the page, the answer can’t cite it.
Allow the ones you want to be found by.
Why it matters
If your robots.txt blocks the search crawlers, those engines cannot read your pages, so their answers cannot cite you.
Do this today
- 1Open yourdomain.co.uk/robots.txt and look for the AI user agents. The ones that decide whether you can be cited are OAI-SearchBot (ChatGPT), Claude-SearchBot (Claude) and PerplexityBot; GPTBot and ClaudeBot are training crawlers, a separate choice.
- 2Decide which you want to be found by and allow them; a blanket 'Disallow: /' blocks them all. Leave Google-Extended alone unless you mean to opt out of Gemini training and grounding: it does not affect AI Overviews, which read what Googlebot fetches.
- 3Use the checker below to confirm what each bot is actually allowed to read.
Read more in the glossary
The same 20-second tip we post as a Short. Plays silently; the button turns the sound on.
The tool
Check which AI bots your robots.txt allows
Enter your domain and we read your robots.txt against the AI crawlers' names. Also at /tools/ai-robots-checker.
We fetch yourdomain/robots.txt once and read it against the AI crawlers' names. Nothing is stored.
What's your AEO score?
Send your domain and a named expert will run it through all six engines, score the results and email your report within 1 to 2 working days. Free, no card, no signup.
- ChatGPT
- Claude
- Gemini
- Perplexity
- Google AI Overviews
- Google AI Mode