How often do firewalls silently block AI search crawlers?
Sample run November 1, 2026
In a random sample of popular sites, about 1 in 28 reachable sites rejected AI search crawler user agents at the firewall in a way aimed at AI crawlers specifically, while robots.txt allowed them.
Sample
600 domains drawn at random from the Tranco top-20,000 list. A site counted as reachable when a normal browser request to its homepage returned HTTP 200 from our server: 362 of 600. The other 238 were excluded because even the browser request didn't get a normal page: 125 failed to connect or timed out (many are CDN, ad or API domains with no public website), 80 returned 403/401/503 or a challenge, and 33 returned another status such as 404. For those, nothing can be said about crawlers specifically.
Step 1: user-agent requests
For each reachable site we fetched robots.txt and requested the homepage with the user agents of four AI search and user-triggered crawlers: OAI-SearchBot, ChatGPT-User, PerplexityBot and Claude-SearchBot. A request counted as rejected when it got a 4xx/5xx status or a challenge page while the browser request got 200.
45 sites (12.4% of reachable) rejected at least one of the four while their robots.txt allowed it. We call this a silent block: nothing in robots.txt warns the site owner. 26 of the 45 were served by Cloudflare.
Step 2: control groups
We re-tested the 45 the same day with two controls: a request using Googlebot's user agent, and a request from an unknown bot. This separates three different causes:
| What the controls showed | Likely cause | Sites |
|---|---|---|
| Googlebot user agent rejected too | The site verifies crawler IP addresses and rejects anyone claiming a crawler name from the wrong network. The real AI crawlers may well get through. | 29 |
| Unknown bot rejected, Googlebot allowed | A general bot rule | 1 |
| Not reproduced on re-test | Rate limiting or a transient rule | 2 |
| Browser, Googlebot and unknown bot allowed; AI search UA rejected | A rule aimed at AI crawlers | 13 |
The 13 are 3.6% of reachable sites, about 1 in 28. Examples include toyota.com, smithsonianmag.com and torontosun.com; 7 of the 13 were served by Cloudflare.
Limits
- All requests came from one data-center server. Real crawlers come from their operators' own IP ranges, which no outside tester can use. A rule that allows verified crawler IPs would let the real crawler through even when our request was rejected; the control groups exist to flag those cases, but can't rule them out completely.
- We tested the homepage only. A site can treat other paths differently.
- Rules change. A result describes the site on November 1, 2026; the links on our homepage run a fresh check.
- 600 is a small sample. Treat 12.4% and 3.6% as estimates with a margin of a few percentage points.
Asian sites
We ran the same test on 1,000 popular Chinese-language sites. Silent blocks were rarer: 3.7% of reachable sites at step 1 and about 0.8% at step 2.
Test your own site
Free, no sign-up. We request your homepage as 13 different clients and show what each one gets.