AI crawler access report

fw-verify-bots-lab.vaksel.com

https://fw-verify-bots-lab.vaksel.com/ · checked 2026-11-02 02:09 UTC · served by Cloudflare

Reachable, with caveats

Nothing is clearly blocked, but some requests were challenged or got a thinner page. See the rows marked “check”.

Watch fw-verify-bots-lab.vaksel.com · get an email if AI crawler access changesEarly access →

Can they get inAAll get in; 8 hit a challenge or a thinner page Can they read itSEvery official and common check passes Can they cite itS5 of 5 citation signals

How we grade

Each layer is graded on its own; there’s no overall score. Only checks marked “Official docs” or “Common practice” count; hypotheses and notes don’t. Blocking training crawlers costs nothing: that’s your call.

Can they get in: S, every AI search crawler got in; A, all get in but some hit a challenge or a thinner page, or the live test didn’t finish and only robots.txt was checked; B, some AI search crawlers are blocked; C, all are blocked, a silent block was found, or the homepage is noindex. A robots.txt that returns 5xx or can’t be reached counts as “disallow everything” under RFC 9309.
Can they read it: S, nothing to improve; A, one item; B, two or three, or a non-critical problem; C, indexing or snippets are blocked, the text needs JavaScript, or four or more items need work.
Can they cite it: five signals (structured data, organization, site name, share card, semantic tags). S all five, A four, B two or three, C one or none; a JSON-LD parse error, or FAQ markup whose questions aren’t on the page, caps it at B.

Rules version 2026-11-02.1 · Full grading method

AI search crawlers reachable

8 / 8

Silent firewall blocks

Training crawlers blocked

0 / 2

Text without JavaScript

990 chars

Crawler by crawler Open Details for the evidence

Crawler	Purpose	robots.txt	Live request	Result
AI search & user-triggered
OAI-SearchBot OpenAI	AI search	Allowed	Verified-bot rule	Check
What we observed GET https://fw-verify-bots-lab.vaksel.com/ with the OAI-SearchBot user agent returned `http_403`. Same URL as a browser: 200 · with a Googlebot user agent: blocked · as an unknown bot: blocked. What it suggests A request claiming to be Googlebot was rejected too, so the site probably verifies crawler IP addresses. The real OAI-SearchBot, from OpenAI’s own network, may well get through. Low confidence. How to confirm In your CDN or WAF logs, look for OAI-SearchBot requests coming from OpenAI’s published IP ranges (linked on the OAI-SearchBot page) and check which rule they hit.
ChatGPT-User OpenAI	User-triggered	Allowed	Verified-bot rule	Check
What we observed GET https://fw-verify-bots-lab.vaksel.com/ with the ChatGPT-User user agent returned `http_403`. Same URL as a browser: 200 · with a Googlebot user agent: blocked · as an unknown bot: blocked. What it suggests A request claiming to be Googlebot was rejected too, so the site probably verifies crawler IP addresses. The real ChatGPT-User, from OpenAI’s own network, may well get through. Low confidence. How to confirm In your CDN or WAF logs, look for ChatGPT-User requests coming from OpenAI’s published IP ranges (linked on the ChatGPT-User page) and check which rule they hit.
PerplexityBot Perplexity	AI search	Allowed	Verified-bot rule	Check
What we observed GET https://fw-verify-bots-lab.vaksel.com/ with the PerplexityBot user agent returned `http_403`. Same URL as a browser: 200 · with a Googlebot user agent: blocked · as an unknown bot: blocked. What it suggests A request claiming to be Googlebot was rejected too, so the site probably verifies crawler IP addresses. The real PerplexityBot, from Perplexity’s own network, may well get through. Low confidence. How to confirm In your CDN or WAF logs, look for PerplexityBot requests coming from Perplexity’s published IP ranges (linked on the PerplexityBot page) and check which rule they hit.
Perplexity-User Perplexity	User-triggered	Allowed	Verified-bot rule	Check
What we observed GET https://fw-verify-bots-lab.vaksel.com/ with the Perplexity-User user agent returned `http_403`. Same URL as a browser: 200 · with a Googlebot user agent: blocked · as an unknown bot: blocked. What it suggests A request claiming to be Googlebot was rejected too, so the site probably verifies crawler IP addresses. The real Perplexity-User, from Perplexity’s own network, may well get through. Low confidence. How to confirm In your CDN or WAF logs, look for Perplexity-User requests coming from Perplexity’s published IP ranges (linked on the Perplexity-User page) and check which rule they hit.
Claude-SearchBot Anthropic	AI search	Allowed	Verified-bot rule	Check
What we observed GET https://fw-verify-bots-lab.vaksel.com/ with the Claude-SearchBot user agent returned `http_403`. Same URL as a browser: 200 · with a Googlebot user agent: blocked · as an unknown bot: blocked. What it suggests A request claiming to be Googlebot was rejected too, so the site probably verifies crawler IP addresses. The real Claude-SearchBot, from Anthropic’s own network, may well get through. Low confidence. How to confirm In your CDN or WAF logs, look for Claude-SearchBot requests coming from Anthropic’s published IP ranges (linked on the Claude-SearchBot page) and check which rule they hit.
Claude-User Anthropic	User-triggered	Allowed	Verified-bot rule	Check
What we observed GET https://fw-verify-bots-lab.vaksel.com/ with the Claude-User user agent returned `http_403`. Same URL as a browser: 200 · with a Googlebot user agent: blocked · as an unknown bot: blocked. What it suggests A request claiming to be Googlebot was rejected too, so the site probably verifies crawler IP addresses. The real Claude-User, from Anthropic’s own network, may well get through. Low confidence. How to confirm In your CDN or WAF logs, look for Claude-User requests coming from Anthropic’s published IP ranges (linked on the Claude-User page) and check which rule they hit.
Bytespider ByteDance	AI search	Allowed	Verified-bot rule	Check
What we observed GET https://fw-verify-bots-lab.vaksel.com/ with the Bytespider user agent returned `http_403`. Same URL as a browser: 200 · with a Googlebot user agent: blocked · as an unknown bot: blocked. What it suggests A request claiming to be Googlebot was rejected too, so the site probably verifies crawler IP addresses. The real Bytespider, from ByteDance’s own network, may well get through. Low confidence. How to confirm In your CDN or WAF logs, look for Bytespider requests coming from ByteDance’s published IP ranges (linked on the Bytespider page) and check which rule they hit.
PetalBot Huawei	AI search	Allowed	Verified-bot rule	Check
What we observed GET https://fw-verify-bots-lab.vaksel.com/ with the PetalBot user agent returned `http_403`. Same URL as a browser: 200 · with a Googlebot user agent: blocked · as an unknown bot: blocked. What it suggests A request claiming to be Googlebot was rejected too, so the site probably verifies crawler IP addresses. The real PetalBot, from Huawei’s own network, may well get through. Low confidence. How to confirm In your CDN or WAF logs, look for PetalBot requests coming from Huawei’s published IP ranges (linked on the PetalBot page) and check which rule they hit.
AI training
GPTBot OpenAI	Training	Allowed	Verified-bot rule	Check
What we observed GET https://fw-verify-bots-lab.vaksel.com/ with the GPTBot user agent returned `http_403`. Same URL as a browser: 200 · with a Googlebot user agent: blocked · as an unknown bot: blocked. What it suggests A request claiming to be Googlebot was rejected too, so the site probably verifies crawler IP addresses. The real GPTBot, from OpenAI’s own network, may well get through. Low confidence. How to confirm In your CDN or WAF logs, look for GPTBot requests coming from OpenAI’s published IP ranges (linked on the GPTBot page) and check which rule they hit.
ClaudeBot Anthropic	Training	Allowed	Verified-bot rule	Check
What we observed GET https://fw-verify-bots-lab.vaksel.com/ with the ClaudeBot user agent returned `http_403`. Same URL as a browser: 200 · with a Googlebot user agent: blocked · as an unknown bot: blocked. What it suggests A request claiming to be Googlebot was rejected too, so the site probably verifies crawler IP addresses. The real ClaudeBot, from Anthropic’s own network, may well get through. Low confidence. How to confirm In your CDN or WAF logs, look for ClaudeBot requests coming from Anthropic’s published IP ranges (linked on the ClaudeBot page) and check which rule they hit.

Can they read it 10 pass · 0 to improve · 0 problems

Whether search engines and AI can parse and index your homepage

Pass
Page title
“VAKSEL Lab: fw-verify-bots · AI visibility test site”.
Official docs ↗
Pass
Meta description
“A single-variable test site from the VAKSEL lab (fw-verify-bots). Each lab site changes on…”.
Official docs ↗
Pass
Indexing and snippets allowed
No noindex or nosnippet.
Official docs ↗
Pass
Canonical URL
Points to https://fw-verify-bots-lab.vaksel.com/.
Official docs ↗
Pass
Text without JavaScript
937 characters are readable without JavaScript.
Official docs ↗
Pass
H1 heading
“VAKSEL Lab: fw-verify-bots”.
Common practice
Pass
Sitemap
Declared in robots.txt: https://fw-verify-bots-lab.vaksel.com/sitemap.xml.
Official docs ↗
Pass
Internal links
6 internal links on the homepage.
Official docs ↗
Pass
Image alt text
1 of 1 images have alt text.
Official docs ↗
Pass
Mobile viewport
Viewport is set.
Official docs ↗

Can they cite it 5 pass · 0 to improve · 0 problems

Signals that help AI recognize and quote you

Pass
Structured data
1 block: Organization, WebSite.
Official docs ↗
Pass
Organization
“VAKSEL Lab: fw-verify-bots”, with logo.
Official docs ↗
Pass
Site name
“VAKSEL Lab: fw-verify-bots”.
Official docs ↗
Pass
Semantic tags
Uses <main> or <article>, so the main content is easy to find.
Common practice
Pass
Share card
Has og:title and og:image.
Common practice
Note
Published and updated dates
No date signals. Homepages rarely need them; articles do.
Official docs ↗
Note
Author
No author on the homepage. Name authors on articles.
Common practice
Note
Question-and-answer sections
No question-style headings. FAQs work well as a question heading followed by a direct answer.
Hypothesis
Note
llms.txt
None. Google says it isn’t needed; other AI providers haven’t said whether they read it. Optional.
Hypothesis

Google says there are no extra requirements for AI Overviews and no special AI files or markup are needed: the page must be indexed and eligible for a snippet. So we list facts you can verify, and anything marked “Hypothesis” is only a suggestion.

How we test, and what it can’t tell you

We request your homepage from a data-center IP as a normal browser, with a Googlebot user agent, as an unknown bot, and with each AI crawler’s user agent. We don’t come from the crawlers’ own networks, so a site that verifies crawler IP addresses may treat the real crawler differently. The control groups separate a rule aimed at AI crawlers (high confidence) from a general bot rule (medium) or an IP-verification rule (low). To be certain, check your CDN or WAF logs for requests from the crawler’s published IP ranges. Methodology

Page checks

Page titlePresent
noindexNo
nosnippetNo
Content without JSOK
robots.txtFound
llms.txt (optional)None

fw-verify-bots-lab.vaksel.com

Reachable, with caveats

Crawler by crawler Open Details for the evidence

What we observed

What it suggests

How to confirm

What we observed

What it suggests

How to confirm

What we observed

What it suggests

How to confirm

What we observed

What it suggests

How to confirm

What we observed

What it suggests

How to confirm

What we observed

What it suggests

How to confirm

What we observed

What it suggests

How to confirm

What we observed

What it suggests

How to confirm

What we observed

What it suggests

How to confirm

What we observed

What it suggests

How to confirm

Can they read it 10 pass · 0 to improve · 0 problems

Can they cite it 5 pass · 0 to improve · 0 problems

How we test, and what it can’t tell you

Page checks

Share this report