AI crawler access report

robots-disallow-all-lab.vaksel.com

https://robots-disallow-all-lab.vaksel.com/ · checked · served by Cloudflare

AI search crawlers are blocked

Every AI search crawler we tested was disallowed by robots.txt or rejected in our live requests. Open a row for the evidence and its confidence.

Watch robots-disallow-all-lab.vaksel.com · get an email if AI crawler access changesEarly access →
Can they get inCAll 8 AI search crawlers blockedCan they read itSEvery official and common check passesCan they cite itS5 of 5 citation signals
How we grade

Each layer is graded on its own; there’s no overall score. Only checks marked “Official docs” or “Common practice” count; hypotheses and notes don’t. Blocking training crawlers costs nothing: that’s your call.

  • Can they get in: S, every AI search crawler got in; A, all get in but some hit a challenge or a thinner page, or the live test didn’t finish and only robots.txt was checked; B, some AI search crawlers are blocked; C, all are blocked, a silent block was found, or the homepage is noindex. A robots.txt that returns 5xx or can’t be reached counts as “disallow everything” under RFC 9309.
  • Can they read it: S, nothing to improve; A, one item; B, two or three, or a non-critical problem; C, indexing or snippets are blocked, the text needs JavaScript, or four or more items need work.
  • Can they cite it: five signals (structured data, organization, site name, share card, semantic tags). S all five, A four, B two or three, C one or none; a JSON-LD parse error, or FAQ markup whose questions aren’t on the page, caps it at B.

Rules version 2026-11-02.1 · Full grading method

AI search crawlers reachable
0 / 8
Silent firewall blocks
0
Training crawlers blocked
2 / 2
Text without JavaScript
1,005 chars

Crawler by crawler Open Details for the evidence

CrawlerPurposerobots.txtLive requestResult
AI search & user-triggered
OAI-SearchBot
OpenAI
AI searchDisallowed200 OK
Blocked
ChatGPT-User
OpenAI
User-triggeredDisallowed200 OK
Blocked
PerplexityBot
Perplexity
AI searchDisallowed200 OK
Blocked
Perplexity-User
Perplexity
User-triggeredDisallowed200 OK
Blocked
Claude-SearchBot
Anthropic
AI searchDisallowed200 OK
Blocked
Claude-User
Anthropic
User-triggeredDisallowed200 OK
Blocked
Bytespider
ByteDance
AI searchDisallowed200 OK
Blocked
PetalBot
Huawei
AI searchDisallowed200 OK
Blocked
AI training
GPTBot
OpenAI
TrainingDisallowed200 OK
Blocked
ClaudeBot
Anthropic
TrainingDisallowed200 OK
Blocked

How to fix

robots.txt disallows OAI-SearchBot, ChatGPT-User, PerplexityBot, Perplexity-User, Claude-SearchBot, Claude-User, Bytespider, PetalBot. To stay out of AI training but appear in AI search, use:

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: OAI-SearchBot
User-agent: PerplexityBot
User-agent: Claude-SearchBot
Allow: /

Can they read it 10 pass · 0 to improve · 0 problems

Whether search engines and AI can parse and index your homepage

  • Pass
    Page title

    “VAKSEL Lab: robots-disallow-all · AI visibility test site”.

    Official docs ↗
  • Pass
    Meta description

    “A single-variable test site from the VAKSEL lab (robots-disallow-all). Each lab site chang…”.

    Official docs ↗
  • Pass
    Indexing and snippets allowed

    No noindex or nosnippet.

    Official docs ↗
  • Pass
    Canonical URL

    Points to https://robots-disallow-all-lab.vaksel.com/.

    Official docs ↗
  • Pass
    Text without JavaScript

    947 characters are readable without JavaScript.

    Official docs ↗
  • Pass
    H1 heading

    “VAKSEL Lab: robots-disallow-all”.

    Common practice
  • Pass
    Sitemap

    Declared in robots.txt: https://robots-disallow-all-lab.vaksel.com/sitemap.xml.

    Official docs ↗
  • Pass
    Internal links

    6 internal links on the homepage.

    Official docs ↗
  • Pass
    Image alt text

    1 of 1 images have alt text.

    Official docs ↗
  • Pass
    Mobile viewport

    Viewport is set.

    Official docs ↗

Can they cite it 5 pass · 0 to improve · 0 problems

Signals that help AI recognize and quote you

  • Pass
    Structured data

    1 block: Organization, WebSite.

    Official docs ↗
  • Pass
    Organization

    “VAKSEL Lab: robots-disallow-all”, with logo.

    Official docs ↗
  • Pass
    Site name

    “VAKSEL Lab: robots-disallow-all”.

    Official docs ↗
  • Pass
    Semantic tags

    Uses <main> or <article>, so the main content is easy to find.

    Common practice
  • Pass
    Share card

    Has og:title and og:image.

    Common practice
  • Note
    Published and updated dates

    No date signals. Homepages rarely need them; articles do.

    Official docs ↗
  • Note
    Author

    No author on the homepage. Name authors on articles.

    Common practice
  • Note
    Question-and-answer sections

    No question-style headings. FAQs work well as a question heading followed by a direct answer.

    Hypothesis
  • Note
    llms.txt

    None. Google says it isn’t needed; other AI providers haven’t said whether they read it. Optional.

    Hypothesis

Google says there are no extra requirements for AI Overviews and no special AI files or markup are needed: the page must be indexed and eligible for a snippet. So we list facts you can verify, and anything marked “Hypothesis” is only a suggestion.

How we test, and what it can’t tell you

We request your homepage from a data-center IP as a normal browser, with a Googlebot user agent, as an unknown bot, and with each AI crawler’s user agent. We don’t come from the crawlers’ own networks, so a site that verifies crawler IP addresses may treat the real crawler differently. The control groups separate a rule aimed at AI crawlers (high confidence) from a general bot rule (medium) or an IP-verification rule (low). To be certain, check your CDN or WAF logs for requests from the crawler’s published IP ranges. Methodology

Page checks

  • Page titlePresent
  • noindexNo
  • nosnippetNo
  • Content without JSOK
  • robots.txtFound
  • llms.txt (optional)None

Share this report