fw-ai-thin-lab.vaksel.com
Reachable, with caveats
Nothing is clearly blocked, but some requests were challenged or got a thinner page. See the rows marked “check”.
How we grade
Each layer is graded on its own; there’s no overall score. Only checks marked “Official docs” or “Common practice” count; hypotheses and notes don’t. Blocking training crawlers costs nothing: that’s your call.
- Can they get in: S, every AI search crawler got in; A, all get in but some hit a challenge or a thinner page, or the live test didn’t finish and only robots.txt was checked; B, some AI search crawlers are blocked; C, all are blocked, a silent block was found, or the homepage is noindex. A robots.txt that returns 5xx or can’t be reached counts as “disallow everything” under RFC 9309.
- Can they read it: S, nothing to improve; A, one item; B, two or three, or a non-critical problem; C, indexing or snippets are blocked, the text needs JavaScript, or four or more items need work.
- Can they cite it: five signals (structured data, organization, site name, share card, semantic tags). S all five, A four, B two or three, C one or none; a JSON-LD parse error, or FAQ markup whose questions aren’t on the page, caps it at B.
Rules version 2026-11-02.1 · Full grading method
Crawler by crawler Open Details for the evidence
| Crawler | robots.txt | Result | ||
|---|---|---|---|---|
| AI search & user-triggered | ||||
OpenAI | Allowed | Check | ||
What we observedGET https://fw-ai-thin-lab.vaksel.com/ with the OAI-SearchBot user agent returned 200, but with far less readable text than a browser got. What it suggestsThe site may serve crawlers a stripped or interstitial page. How to confirmFetch the page with the OAI-SearchBot user agent yourself (for example with curl -A) and compare it with what a browser shows. | ||||
OpenAI | Allowed | Check | ||
What we observedGET https://fw-ai-thin-lab.vaksel.com/ with the ChatGPT-User user agent returned 200, but with far less readable text than a browser got. What it suggestsThe site may serve crawlers a stripped or interstitial page. How to confirmFetch the page with the ChatGPT-User user agent yourself (for example with curl -A) and compare it with what a browser shows. | ||||
Perplexity | Allowed | Check | ||
What we observedGET https://fw-ai-thin-lab.vaksel.com/ with the PerplexityBot user agent returned 200, but with far less readable text than a browser got. What it suggestsThe site may serve crawlers a stripped or interstitial page. How to confirmFetch the page with the PerplexityBot user agent yourself (for example with curl -A) and compare it with what a browser shows. | ||||
Perplexity | Allowed | Check | ||
What we observedGET https://fw-ai-thin-lab.vaksel.com/ with the Perplexity-User user agent returned 200, but with far less readable text than a browser got. What it suggestsThe site may serve crawlers a stripped or interstitial page. How to confirmFetch the page with the Perplexity-User user agent yourself (for example with curl -A) and compare it with what a browser shows. | ||||
Anthropic | Allowed | Check | ||
What we observedGET https://fw-ai-thin-lab.vaksel.com/ with the Claude-SearchBot user agent returned 200, but with far less readable text than a browser got. What it suggestsThe site may serve crawlers a stripped or interstitial page. How to confirmFetch the page with the Claude-SearchBot user agent yourself (for example with curl -A) and compare it with what a browser shows. | ||||
Anthropic | Allowed | Check | ||
What we observedGET https://fw-ai-thin-lab.vaksel.com/ with the Claude-User user agent returned 200, but with far less readable text than a browser got. What it suggestsThe site may serve crawlers a stripped or interstitial page. How to confirmFetch the page with the Claude-User user agent yourself (for example with curl -A) and compare it with what a browser shows. | ||||
ByteDance | Allowed | Reachable | ||
What we observedGET https://fw-ai-thin-lab.vaksel.com/ with the Bytespider user agent got the same page a browser gets. robots.txt: Allowed. What it suggestsNo sign of a block for this crawler. | ||||
Huawei | Allowed | Reachable | ||
What we observedGET https://fw-ai-thin-lab.vaksel.com/ with the PetalBot user agent got the same page a browser gets. robots.txt: Allowed. What it suggestsNo sign of a block for this crawler. | ||||
| AI training | ||||
OpenAI | Allowed | Reachable | ||
What we observedGET https://fw-ai-thin-lab.vaksel.com/ with the GPTBot user agent got the same page a browser gets. robots.txt: Allowed. What it suggestsNo sign of a block for this crawler. | ||||
Anthropic | Allowed | Reachable | ||
What we observedGET https://fw-ai-thin-lab.vaksel.com/ with the ClaudeBot user agent got the same page a browser gets. robots.txt: Allowed. What it suggestsNo sign of a block for this crawler. | ||||
Can they read it 10 pass · 0 to improve · 0 problems
Whether search engines and AI can parse and index your homepage
- PassPage titleOfficial docs ↗
“VAKSEL Lab: fw-ai-thin · AI visibility test site”.
- PassMeta descriptionOfficial docs ↗
“A single-variable test site from the VAKSEL lab (fw-ai-thin). Each lab site changes one th…”.
- PassIndexing and snippets allowedOfficial docs ↗
No noindex or nosnippet.
- PassCanonical URLOfficial docs ↗
Points to https://fw-ai-thin-lab.vaksel.com/.
- PassText without JavaScriptOfficial docs ↗
929 characters are readable without JavaScript.
- PassH1 headingCommon practice
“VAKSEL Lab: fw-ai-thin”.
- PassSitemapOfficial docs ↗
Declared in robots.txt: https://fw-ai-thin-lab.vaksel.com/sitemap.xml.
- PassInternal linksOfficial docs ↗
6 internal links on the homepage.
- PassImage alt textOfficial docs ↗
1 of 1 images have alt text.
- PassMobile viewportOfficial docs ↗
Viewport is set.
Can they cite it 5 pass · 0 to improve · 0 problems
Signals that help AI recognize and quote you
- PassStructured dataOfficial docs ↗
1 block: Organization, WebSite.
- PassOrganizationOfficial docs ↗
“VAKSEL Lab: fw-ai-thin”, with logo.
- PassSite nameOfficial docs ↗
“VAKSEL Lab: fw-ai-thin”.
- PassSemantic tagsCommon practice
Uses <main> or <article>, so the main content is easy to find.
- PassShare cardCommon practice
Has og:title and og:image.
- NotePublished and updated datesOfficial docs ↗
No date signals. Homepages rarely need them; articles do.
- NoteAuthorCommon practice
No author on the homepage. Name authors on articles.
- NoteQuestion-and-answer sectionsHypothesis
No question-style headings. FAQs work well as a question heading followed by a direct answer.
- Notellms.txtHypothesis
None. Google says it isn’t needed; other AI providers haven’t said whether they read it. Optional.
Google says there are no extra requirements for AI Overviews and no special AI files or markup are needed: the page must be indexed and eligible for a snippet. So we list facts you can verify, and anything marked “Hypothesis” is only a suggestion.
How we test, and what it can’t tell you
We request your homepage from a data-center IP as a normal browser, with a Googlebot user agent, as an unknown bot, and with each AI crawler’s user agent. We don’t come from the crawlers’ own networks, so a site that verifies crawler IP addresses may treat the real crawler differently. The control groups separate a rule aimed at AI crawlers (high confidence) from a general bot rule (medium) or an IP-verification rule (low). To be certain, check your CDN or WAF logs for requests from the crawler’s published IP ranges. Methodology
Page checks
- Page titlePresent
- noindexNo
- nosnippetNo
- Content without JSOK
- robots.txtFound
- llms.txt (optional)None