WordPress site invisible to ChatGPT or Perplexity? Five places to check
Updated 2026-11-01
On WordPress, an AI crawler can be shut out at five different layers. Work down this list; the report tells you whether the block is in robots.txt or in a live request.
1. “Discourage search engines from indexing this site”
In Settings → Reading, this checkbox asks search engines not to index the site. Staging sites often go live with it still on. Untick it if the report shows noindex.
2. Your SEO plugin's robots.txt editor
WordPress serves a virtual robots.txt, and SEO plugins such as Yoast SEO and Rank Math let you edit it from the dashboard. If the report says robots.txt disallows an AI search crawler, look there first, or check for a physical robots.txt file in your web root, which overrides the virtual one.
To opt out of training but stay in AI search, keep the training crawlers blocked and the search crawlers allowed:
User-agent: GPTBot Disallow: / User-agent: ClaudeBot Disallow: / User-agent: Google-Extended Disallow: / User-agent: OAI-SearchBot User-agent: PerplexityBot User-agent: Claude-SearchBot Allow: /
3. Security plugins
Wordfence, Sucuri, All-In-One Security and similar plugins can block by user agent, rate-limit crawlers, or block “fake” bots. If the live request is blocked but robots.txt allows it, check their blocklists and bot rules for AI user agents.
4. “Block AI” plugins
Some plugins add every AI agent to robots.txt or block them outright, without separating search from training. If you use one, check whether it lets you allow search crawlers separately.
5. Your host or CDN
Managed WordPress hosts often run their own bot protection, and many sites sit behind Cloudflare. If the report says the request was answered by a CDN, follow the Cloudflare guide, or ask your host whether they block AI user agents at the edge.
Sources
Re-test your site
Free, no sign-up. We request your homepage as 13 different clients and show what each one gets.
Other guides: Cloudflare · CloudFront / AWS WAF · Shopify