Cloudflare is blocking AI crawlers: how to fix it (2026 controls)

Updated 2026-11-01

Cloudflare replaced its single “Block AI bots” switch with three controls on September 15, 2026. A setting that used to be harmless can now block search crawlers, including Googlebot, Bingbot and Applebot.

The short version. In Security → Settings, set Search to Allow, Training to Disallow AI Training, and Agent to whatever you prefer. Don't use Block for Training unless you're happy to leave Google, Bing and Apple search too.

1. Check the three AI bot controls

Cloudflare now classifies bots by behavior. Each control has its own setting:

Control	What it covers	Setting we recommend
Search	Crawlers that index content to answer questions later, such as OAI-SearchBot, PerplexityBot and Claude-SearchBot.	Allow
Training	Crawlers that collect content to train models, including mixed-use crawlers that do both search and training.	Disallow AI Training
Agent	User-directed fetches, such as ChatGPT-User, Claude-User and browser-use agents.	Allow, or Block on pages with ads

Options are Allow, Block on pages with ads and Block; Training also offers Disallow AI Training.

2. Understand what “Block” means now

From September 15, 2026, Block and Block on pages with ads apply to mixed-use crawlers too. Cloudflare's own announcement says selecting Block for Training “will stop Applebot, Bingbot, and Googlebot from reaching your site, search included.”

Disallow AI Training is the search-safe option: Cloudflare publishes a no-training preference in your robots.txt, keeps crawlers it designates “Accountable” (Apple, Google, Microsoft) allowed for search, and blocks training-only crawlers such as GPTBot, ClaudeBot, Meta-ExternalAgent and Amazonbot.

3. Check what Cloudflare migrated for you

Existing sites were migrated automatically. If you had the old Block AI bots switch on, you now have Search Allow, Training Disallow AI Training, Agent Block on pages with ads. New domains on ad-supported sites get the same preset. That Agent setting can block ChatGPT-User and Claude-User on pages that show ads.

4. Look for custom rules and Bot Fight Mode

Security rules / WAF custom rules: search for expressions using cf.verified_bot_category, http.user_agent or bot names like GPTBot. An old rule written before the new controls may still block AI search crawlers.
Bot Fight Mode / Super Bot Fight Mode: these can challenge automated traffic. If AI crawlers get challenge pages, review these settings and allow verified bots.
AI Crawl Control (formerly AI Audit): shows which AI crawlers hit your site and lets you allow or block individual crawlers. Check that the search crawlers you want aren't set to Block there.

5. Re-test

Cloudflare settings apply within seconds. Run the check again and confirm the AI search rows say Reachable.

Sources

Re-test your site

Free, no sign-up. We request your homepage as 13 different clients and show what each one gets.

Other guides: CloudFront / AWS WAF · WordPress · Shopify