Free · no sign-up · results in seconds

Can AI search actually reach your site?

Your robots.txt may say yes while your firewall says no. We request your page with the user agents of ChatGPT, Perplexity and Claude, compare against a browser and Googlebot, and show you what each one got back.

Try ··

vaksel.com/report/toyota.com

AI crawler access report

toyota.com

https://www.toyota.com/ · checked 2026-11-02 18:38 UTC · served by CloudFront

Silent block: requests as AI search crawlers are rejected

robots.txt says they’re welcome, but our requests with their user agents were rejected while a browser and a Googlebot request got through. Open a row for what we saw and how to confirm it.

Watch toyota.com · get an email if AI crawler access changesEarly access →

Can they get inC4 AI search crawlers silently blocked by the firewall Can they read itATo improve: H1 heading Can they cite itB3 of 5 citation signals; missing Organization, Site name

China search & AI engines (not graded): 6 of 7 can get in. Blocked: Bytespider

How we grade

Each layer is graded on its own; there’s no overall score. Only checks marked “Official docs” or “Common practice” count; hypotheses and notes don’t. Blocking training crawlers costs nothing: that’s your call.

Can they get in: S, every AI search crawler got in; A, all get in but some hit a challenge or a thinner page, or the live test didn’t finish and only robots.txt was checked; B, some AI search crawlers are blocked; C, all are blocked, a silent block was found, or the homepage is noindex. A robots.txt that returns 5xx or can’t be reached counts as “disallow everything” under RFC 9309.
Can they read it: S, nothing to improve; A, one item; B, two or three, or a non-critical problem; C, indexing or snippets are blocked, the text needs JavaScript, or four or more items need work.
Can they cite it: five signals (structured data, organization, site name, share card, semantic tags). S all five, A four, B two or three, C one or none; a JSON-LD parse error, or FAQ markup whose questions aren’t on the page, caps it at B.

Rules version 2026-11-02.2 · Full grading method

AI search crawlers reachable

1 / 6

Silent firewall blocks

Training crawlers blocked

2 / 2

Text without JavaScript

15,372 chars

Crawler by crawler Open Details for the evidence

Crawler	Purpose	robots.txt	Live request	Result
AI search & user-triggered
OAI-SearchBot OpenAI	AI search	Partly	403	Silent block
What we observed GET https://www.toyota.com/ with the OAI-SearchBot user agent returned `http_403`. Same URL as a browser: 200 · with a Googlebot user agent: 200 · as an unknown bot: 200. What it suggests The rejection follows the OAI-SearchBot user agent, not bots in general: most likely a rule aimed at AI crawlers. High confidence. How to confirm In your CDN or WAF logs, look for OAI-SearchBot requests coming from OpenAI’s published IP ranges (linked on the OAI-SearchBot page) and check which rule they hit.
ChatGPT-User OpenAI	User-triggered	Partly	403	Silent block
What we observed GET https://www.toyota.com/ with the ChatGPT-User user agent returned `http_403`. Same URL as a browser: 200 · with a Googlebot user agent: 200 · as an unknown bot: 200. What it suggests The rejection follows the ChatGPT-User user agent, not bots in general: most likely a rule aimed at AI crawlers. High confidence. How to confirm In your CDN or WAF logs, look for ChatGPT-User requests coming from OpenAI’s published IP ranges (linked on the ChatGPT-User page) and check which rule they hit.
PerplexityBot Perplexity	AI search	Disallowed	403	Blocked
What we observed GET https://www.toyota.com/ with the PerplexityBot user agent returned `http_403`. Same URL as a browser: 200 · with a Googlebot user agent: 200 · as an unknown bot: 200. What it suggests The rejection follows the PerplexityBot user agent, not bots in general: most likely a rule aimed at AI crawlers. High confidence. How to confirm In your CDN or WAF logs, look for PerplexityBot requests coming from Perplexity’s published IP ranges (linked on the PerplexityBot page) and check which rule they hit.
Perplexity-User Perplexity	User-triggered	Partly	403	Silent block
What we observed GET https://www.toyota.com/ with the Perplexity-User user agent returned `http_403`. Same URL as a browser: 200 · with a Googlebot user agent: 200 · as an unknown bot: 200. What it suggests The rejection follows the Perplexity-User user agent, not bots in general: most likely a rule aimed at AI crawlers. High confidence. How to confirm In your CDN or WAF logs, look for Perplexity-User requests coming from Perplexity’s published IP ranges (linked on the Perplexity-User page) and check which rule they hit.
Claude-SearchBot Anthropic	AI search	Partly	200 OK	Reachable
What we observed GET https://www.toyota.com/ with the Claude-SearchBot user agent got the same page a browser gets. robots.txt: Partly. What it suggests No sign of a block for this crawler.
Claude-User Anthropic	User-triggered	Partly	403	Silent block
What we observed GET https://www.toyota.com/ with the Claude-User user agent returned `http_403`. Same URL as a browser: 200 · with a Googlebot user agent: 200 · as an unknown bot: 200. What it suggests The rejection follows the Claude-User user agent, not bots in general: most likely a rule aimed at AI crawlers. High confidence. How to confirm In your CDN or WAF logs, look for Claude-User requests coming from Anthropic’s published IP ranges (linked on the Claude-User page) and check which rule they hit.
China search & AI engines (not graded)
Bytespider ByteDance	AI search	Disallowed	403	Blocked
What we observed GET https://www.toyota.com/ with the Bytespider user agent returned `http_403`. Same URL as a browser: 200 · with a Googlebot user agent: 200 · as an unknown bot: 200. What it suggests The rejection follows the Bytespider user agent, not bots in general: most likely a rule aimed at AI crawlers. High confidence. How to confirm In your CDN or WAF logs, look for Bytespider requests coming from ByteDance’s published IP ranges (linked on the Bytespider page) and check which rule they hit.
PetalBot Huawei	AI search	Partly	200 OK	Reachable
What we observed GET https://www.toyota.com/ with the PetalBot user agent got the same page a browser gets. robots.txt: Partly. What it suggests No sign of a block for this crawler.
Sogou web spider 搜狗（腾讯）	AI search	Partly	200 OK	Reachable
What we observed GET https://www.toyota.com/ with the Sogou web spider user agent got the same page a browser gets. robots.txt: Partly. What it suggests No sign of a block for this crawler.
360Spider 360	AI search	Partly	200 OK	Reachable
What we observed GET https://www.toyota.com/ with the 360Spider user agent got the same page a browser gets. robots.txt: Partly. What it suggests No sign of a block for this crawler.
YisouSpider 神马搜索（阿里）	AI search	Partly	200 OK	Reachable
What we observed GET https://www.toyota.com/ with the YisouSpider user agent got the same page a browser gets. robots.txt: Partly. What it suggests No sign of a block for this crawler.
Kimi-SearchBot 月之暗面 Kimi	AI search	Partly	200 OK	Reachable
What we observed GET https://www.toyota.com/ with the Kimi-SearchBot user agent got the same page a browser gets. robots.txt: Partly. What it suggests No sign of a block for this crawler.
Kimi-User 月之暗面 Kimi	User-triggered	Partly	200 OK	Reachable
What we observed GET https://www.toyota.com/ with the Kimi-User user agent got the same page a browser gets. robots.txt: Partly. What it suggests No sign of a block for this crawler.
AI training
GPTBot OpenAI	Training	Disallowed	403	Blocked
What we observed GET https://www.toyota.com/ with the GPTBot user agent returned `http_403`. Same URL as a browser: 200 · with a Googlebot user agent: 200 · as an unknown bot: 200. What it suggests The rejection follows the GPTBot user agent, not bots in general: most likely a rule aimed at AI crawlers. High confidence. How to confirm In your CDN or WAF logs, look for GPTBot requests coming from OpenAI’s published IP ranges (linked on the GPTBot page) and check which rule they hit.
ClaudeBot Anthropic	Training	Disallowed	403	Blocked
What we observed GET https://www.toyota.com/ with the ClaudeBot user agent returned `http_403`. Same URL as a browser: 200 · with a Googlebot user agent: 200 · as an unknown bot: 200. What it suggests The rejection follows the ClaudeBot user agent, not bots in general: most likely a rule aimed at AI crawlers. High confidence. How to confirm In your CDN or WAF logs, look for ClaudeBot requests coming from Anthropic’s published IP ranges (linked on the ClaudeBot page) and check which rule they hit.

How to fix

Amazon CloudFront is answering these requests, most likely through AWS WAF. The Bot Control rule CategoryAI blocks AI crawlers even when they are verified. Override it to Count, then re-block only the training crawlers with your own rule.

Crawlers to allow for AI search: OAI-SearchBot, ChatGPT-User, PerplexityBot, Perplexity-User, Claude-User, Bytespider.

robots.txt disallows PerplexityBot. To stay out of AI training but appear in AI search, use:

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: OAI-SearchBot
User-agent: PerplexityBot
User-agent: Claude-SearchBot
Allow: /

Step-by-step CloudFront guide →

Can they read it 10 pass · 1 to improve · 0 problems

Whether search engines and AI can parse and index your homepage

Improve
H1 heading
No <h1>.
Common practice
Pass
Page title
“New Cars, Trucks, SUVs & Hybrids | Toyota Official Site”.
Official docs ↗
Pass
Meta description
“Explore the newest Toyota trucks, cars, SUVs, hybrids and minivans. See photos, compare mo…”.
Official docs ↗
Pass
Indexing and snippets allowed
No noindex or nosnippet.
Official docs ↗
Pass
Canonical URL
Points to https://www.toyota.com/.
Official docs ↗
Pass
Text without JavaScript
15,372 characters are readable without JavaScript.
Official docs ↗
Pass
Sitemap
Declared in robots.txt: https://www.toyota.com/sitemap_index.xml.
Official docs ↗
Pass
Internal links
122 internal links on the homepage.
Official docs ↗
Pass
Image alt text
77 of 77 images have alt text.
Official docs ↗
Pass
hreflang
3 links, with x-default.
Official docs ↗
Pass
Mobile viewport
Viewport is set.
Official docs ↗

Can they cite it 3 pass · 2 to improve · 0 problems

Signals that help AI recognize and quote you

Improve
Organization
No Organization markup. Name, URL and logo help engines tie the site to your brand.
Official docs ↗
Improve
Site name
No WebSite markup or og:site_name, so engines guess your site name.
Official docs ↗
Pass
Structured data
1 block: WebPage.
Official docs ↗
Pass
Semantic tags
Uses <main> or <article>, so the main content is easy to find.
Common practice
Pass
Share card
Has og:title and og:image.
Common practice
Note
Published and updated dates
No date signals. Homepages rarely need them; articles do.
Official docs ↗
Note
Author
No author on the homepage. Name authors on articles.
Common practice
Note
Question-and-answer sections
No question-style headings. FAQs work well as a question heading followed by a direct answer.
Hypothesis
Note
llms.txt
None. Google says it isn’t needed; other AI providers haven’t said whether they read it. Optional.
Hypothesis

Google says there are no extra requirements for AI Overviews and no special AI files or markup are needed: the page must be indexed and eligible for a snippet. So we list facts you can verify, and anything marked “Hypothesis” is only a suggestion.

How we test, and what it can’t tell you

We request your homepage from a data-center IP as a normal browser, with a Googlebot user agent, as an unknown bot, and with each AI crawler’s user agent. We don’t come from the crawlers’ own networks, so a site that verifies crawler IP addresses may treat the real crawler differently. The control groups separate a rule aimed at AI crawlers (high confidence) from a general bot rule (medium) or an IP-verification rule (low). To be certain, check your CDN or WAF logs for requests from the crawler’s published IP ranges. Methodology

Page checks

Page titlePresent
noindexNo
nosnippetNo
Content without JSOK
robots.txtFound
llms.txt (optional)None

Share this report

Open the live report →

A real report from November 3, 2026. Toyota's robots.txt lets OAI-SearchBot in; its firewall turns it away.

Three layers, tested for real

Most checkers only read robots.txt. We also send live requests, with control groups so a bot-verification rule isn't mistaken for a block.

robots.txt rules

A verdict for each of 20+ AI agents, with search and training crawlers kept apart, so you can opt out of training without leaving AI answers.

Firewall & CDN

Live requests as OAI-SearchBot, PerplexityBot, Claude-SearchBot and more. We name the CDN that answered: Cloudflare, CloudFront, Akamai and others.

Page readability

noindex and nosnippet, and whether your content exists without JavaScript, which most AI crawlers don't run.

Our November 2026 sample of popular sites

1 in 28

reachable sites silently block AI search crawlers at the firewall, while robots.txt allows them and Googlebot gets through.

13 of 362 reachable sites, from a random 600 of the Tranco top 20k. Requests sent from a data-center IP with crawler user agents and control groups.

Found in our sample

toyota.comSilent block
CloudFront rejects OAI-SearchBot and three user-triggered crawlers that robots.txt allows. A browser and Googlebot get through.
smithsonianmag.comAll blocked
Cloudflare turns away all 6 AI search and user-triggered crawlers we sent.
torontosun.comSilent block
OpenAI and Perplexity crawlers get a 403 while robots.txt lets them in. Claude’s get through.

Results from November 1, 2026. Each link runs a fresh check. How we measured this →

Fix it where the block is

Each report names the layer that rejected the crawler and links the matching guide.

Cloudflare

Since September 15, 2026 Cloudflare has separate Search, Training and Agent controls.

CloudFront / AWS WAF

AWS WAF Bot Control's CategoryAI rule blocks verified AI crawlers too.

WordPress

A WordPress setting, an SEO plugin, a security plugin or your host can each block AI crawlers.

Shopify

How to change which AI crawlers can access a Shopify store with robots.txt.liquid, keep product pages in AI search, and what to do when the block isn't yours.

Rules change. Get told when they do.

Daily re-checks and an email when a CDN update, plugin or robots.txt edit cuts off an AI crawler. Agencies with many client sites: talk to us.

See plans

Questions

Isn’t robots.txt enough?

robots.txt is only a request. Your CDN, firewall or a security plugin decides what actually gets through, and it can reject a crawler that robots.txt welcomes. That is the silent block we test for.

Will allowing AI search let my content be used for training?

No. OpenAI and Anthropic use separate crawlers for search and training, and Apple and Google give training its own robots.txt name (Applebot-Extended, Google-Extended). Perplexity says PerplexityBot isn’t used to train models. So you can allow OAI-SearchBot and block GPTBot, for example. Our guides show how on each platform.

How accurate is a test from a data center?

Real crawlers come from their own IP ranges, which we can’t use. So every test runs with control groups (a normal browser, Googlebot and an unknown bot) and each result carries a confidence level.

toyota.com

Silent block: requests as AI search crawlers are rejected

Crawler by crawler Open Details for the evidence

What we observed

What it suggests

How to confirm

What we observed

What it suggests

How to confirm

What we observed

What it suggests

How to confirm

What we observed

What it suggests

How to confirm

What we observed

What it suggests

What we observed

What it suggests

How to confirm

What we observed

What it suggests

How to confirm

What we observed

What it suggests

What we observed

What it suggests

What we observed

What it suggests

What we observed

What it suggests

What we observed

What it suggests

What we observed

What it suggests

What we observed

What it suggests

How to confirm

What we observed

What it suggests

How to confirm

How to fix

Can they read it 10 pass · 1 to improve · 0 problems

Can they cite it 3 pass · 2 to improve · 0 problems

How we test, and what it can’t tell you

Page checks

Share this report

Three layers, tested for real

robots.txt rules

Firewall & CDN

Page readability

Found in our sample

Fix it where the block is

Rules change. Get told when they do.

Questions