AEOscoreFree AEO score

Guides / Cloudflare and AI crawlers

Is Cloudflare blocking AI from reading your site? How to check and fix it

By Olly, founder of AEOscore, AEO consultant and full-stack developer, 20+ years in search9 min read

Very likely because a firewall is turning AI crawlers away before robots.txt is ever read. Cloudflare has blocked AI crawlers by default for new domains since 1 July 2025, and the block happens at the network edge, so your robots.txt can say allow while every AI fetch gets a 403. Run a live fetch test to see it, then set your policy in Cloudflare's AI Crawl Control.

On this page

The mechanism

robots.txt is a request, not a lock

Your robots.txt and your firewall are two different doors, and only one of them can actually be locked.

robots.txt is a text file that polite crawlers fetch and choose to obey. A firewall such as Cloudflare sits in front of your whole site and decides which connections get through at all. When Cloudflare turns an AI crawler away, the crawler never reaches your pages, and often never reads your robots.txt either. Every robots.txt checker on the internet, including ours, will happily report “allowed” while the firewall returns 403 to every AI fetch.

That gap is why a site can rank well on Google, pass every robots.txt check, and still never be cited by ChatGPT, Claude or Perplexity: the engines’ search crawlers (OpenAI’s OAI-SearchBot, Anthropic’s Claude-SearchBot, Perplexity’s PerplexityBot) simply cannot read it.

The timeline

What Cloudflare changed, and when

Two announcements, a year apart, and the second one changes the defaults again on 15 September 2026.

On 1 July 2025, Cloudflareannounced it was “changing the default to block AI crawlers unless they pay creators for their content”. Every new domain signing up since then has been asked whether to allow or block AI crawlers, with block as the default posture. Existing zones kept their settings. If your site moved onto Cloudflare after that date, there is a fair chance AI crawlers have been blocked since day one without anyone deciding it on purpose.

On 1 July 2026, Cloudflarerefined the policy into three categories: Search (crawlers that index your content so an engine “can answer questions about it later”), Agent (fetches made on a user’s behalf) and Training (crawlers gathering model training data). From 15 September 2026, the stated defaults are that Training and Agent bots are blocked by default on pages that display ads, while Search crawlers remain allowed by default. The category controls appear in the dashboard under Security settings Cloudflare Docs.

What Cloudflare does to your AI traffic depends on when you signed up and which toggles are set. Check rather than assume.

Diagnose

Check your site in two minutes

Run the checker below: step 1 reads your robots.txt, step 2 fetches your homepage as each AI crawler and compares.

If robots.txt says allowed but the live test shows crawler fetches turned away while a normal browser gets through, something at the server is filtering by bot identity. The tell-tale signs of Cloudflare doing it: a 403 or 402 response Cloudflare Docs, a challenge page carrying the cf-mitigated: challenge header Cloudflare Docs, or error 1010 Cloudflare Docs.

We fetch yourdomain/robots.txt once and read it against the AI crawlers' names. Nothing is stored.

Fix

The two Cloudflare settings that matter

Both live in the Cloudflare dashboard, and both act independently of robots.txt.

1. AI Crawl Control. Select your account and domain, then open AI Crawl Control. It lists AI crawlers individually with three actions: Allow, Block, and Charge for crawl (a paid-access beta). Blocked crawlers can be served a 403, or a 402 Payment Required if you want to signal a licensing route Cloudflare Docs. On the Free plan, crawlers are identified by user-agent string; paid plans add stronger detection.

2. Security settings: Block AI bots. Under Security, the AI bot setting offers Block on all pages, Block on pages with ads, or Allow, applied to the Search, Agent and Training categories Cloudflare Docs.

Which crawlers to allow is your call, and the useful split is by job, not by company. If you want AI engines to cite you, the crawlers that matter are the search ones: OAI-SearchBot, Claude-SearchBot, PerplexityBot and Googlebot. Training crawlers (GPTBot, ClaudeBot) are a separate decision about model training, and blocking them does not remove you from ChatGPT’s or Claude’s live search answers OpenAI. Our robots.txt tip and the glossary cover each bot’s role.

Verify

Verify it worked (and one caveat)

After changing a setting, re-run the live test above and watch the crawler rows flip to readable.

The caveat: Cloudflare verifies genuine crawlers by cryptographic signature or published IP list, not by user-agent string Cloudflare Docs. Our live test sends each bot’s user-agent from our own server, so on some configurations we get challenged where the real, IP-verified crawler would be waved through. Read a “turned away” as proof your server filters by bot identity and a prompt to check the settings above, not as final proof the real crawler is refused. A clean pass the other way is stronger: if our probes get through, there is no user-agent-based block in play.

The definitive test is the engines themselves: ask ChatGPT, Claude and Perplexity a question your buyers ask and see whether your pages can now be cited. Fixes typically show in answers over four to twelve weeks, which is why we track share of voice weekly rather than checking once.

FAQ

Cloudflare and AI crawlers, asked directly

Does robots.txt override Cloudflare?
No. robots.txt is a request that well-behaved crawlers read and follow. Cloudflare acts earlier, at the network edge, and can refuse the connection whatever your robots.txt says. If the two disagree, the firewall wins.
Will allowing AI crawlers hurt my Google rankings?
Google says the Google-Extended control does not affect a site's inclusion in Google Search and is not a ranking signal, and Cloudflare's AI controls are separate from normal search crawling. Allowing AI search crawlers is a visibility decision, not a rankings one.
What does Cloudflare error 1010 mean?
Cloudflare's documentation says error 1010 means the site owner has banned your access based on your browser's signature. If an AI crawler is seeing it on your site, something in your Cloudflare configuration is filtering by client identity.
Is blocking AI crawlers a mistake?
Not necessarily: it is a legitimate choice, and Cloudflare built the controls because many publishers want them. The mistake is blocking by accident. Decide separately about training crawlers (a licensing and rights question) and search crawlers (a visibility question), and set each deliberately.

Sources

Sources

  1. Cloudflare, Content Independence Day: no AI crawl without compensation. Accessed 21 August 2026. Published 1 July 2025. Announces block-by-default for AI crawlers on new domains.
  2. Cloudflare, Your site, your rules: new AI traffic options for all customers. Accessed 21 August 2026. Published 1 July 2026. Introduces the Search, Agent and Training categories and the defaults that apply from 15 September 2026.
  3. Cloudflare Docs, Manage AI crawlers (AI Crawl Control). Accessed 21 August 2026. Per-crawler Allow, Block and Charge for crawl actions; blocked crawlers can be served 403 or 402.
  4. Cloudflare Docs, Block AI bots (Security settings). Accessed 21 August 2026. The Block on all pages, Block on pages with ads, and Allow options, with the category definitions.
  5. Cloudflare Docs, Verified bots. Accessed 21 August 2026. Bots are verified by cryptographic signature or published IP list, not by user-agent string.
  6. Cloudflare Docs, Detect a challenge page response. Accessed 21 August 2026. Challenge responses carry the cf-mitigated: challenge header.
  7. Cloudflare Docs, Error 1010. Accessed 21 August 2026.
  8. OpenAI, OpenAI crawlers and bots. Accessed 21 August 2026. OAI-SearchBot surfaces sites in ChatGPT search; GPTBot is the training crawler; ChatGPT-User fetches for user actions.
  9. Anthropic, Does Anthropic crawl data from the web?. Accessed 21 August 2026. Page dated 7 April 2026. ClaudeBot, Claude-SearchBot and Claude-User all honour robots.txt.
  10. Perplexity, Perplexity crawlers. Accessed 21 August 2026. PerplexityBot powers search citations; Perplexity-User fetches on a user's request and generally ignores robots.txt.

£395 one-off AI Visibility Audit, credited against your first month, £99 a month Monitor, £1,495 a month Monitor + Strategist. Prices exclude VAT.

What changed on this page

  • : Guide published: the 2025 default block, the 2026 category defaults, the live access test, both dashboard settings, and the verified-bots caveat.

Blocked crawlers are one gap of many

The free AEO score checks how all six engines answer the questions your buyers ask, crawler access included, human-checked within 1 to 2 working days. Domain and email, no card.

Get your free AEO scoreSee pricing
  • ChatGPT
  • Claude
  • Gemini
  • Perplexity
  • Google AI Overviews
  • Google AI Mode