Home / Fix guides / CloudFront / AWS WAF

CloudFront or AWS WAF is blocking AI crawlers: how to fix it

Updated 2026-11-01

If your report says CloudFront is answering with 403s to AI crawlers, the usual cause is the AWS WAF Bot Control managed rule group on the web ACL attached to your distribution.

Why it happens. Most Bot Control category rules only block unverified bots. CategoryAI is different: AWS documents that it “applies the action to all matches, regardless of whether the bots are verified or unverified.” So OAI-SearchBot and PerplexityBot get blocked along with training crawlers.

1. Find the rule

  1. Open AWS WAF → Web ACLs (or protection packs), and pick the one associated with your CloudFront distribution.
  2. Find the managed rule group AWSManagedRulesBotControlRuleSet.
  3. In Sampled requests or your WAF logs, look for matches on CategoryAI with the label awswaf:managed:aws:bot-control:bot:category:ai.

2. Switch CategoryAI to Count

Edit the rule group and set a rule action override for CategoryAI to Count. Requests still get labeled, but nothing is blocked by this rule.

3. Re-add the block you actually want

Add your own rule after the Bot Control group that matches the label awswaf:managed:aws:bot-control:bot:category:ai, and exclude the crawlers you want to allow. Use the bot name labels that appear in your logs (awswaf:managed:aws:bot-control:bot:name:…) for OAI-SearchBot, PerplexityBot, Claude-SearchBot and the other search agents. Training crawlers such as GPTBot and ClaudeBot stay blocked.

If you'd rather not maintain a WAF rule, leave CategoryAI on Count and opt out of training with robots.txt instead:

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /

4. Watch the other signals

SignalKnownBotDataCenter and SignalNonBrowserUserAgent only act on requests that aren't from verified bots. If real crawlers are getting through but our data-center test still shows a block, the report marks it as lower confidence for that reason.

5. Re-test

WAF changes take effect within a minute or so. Run the check again.

Sources

Re-test your site

Free, no sign-up. We request your homepage as 13 different clients and show what each one gets.

Other guides: Cloudflare · WordPress · Shopify