meta-webindexer
Meta's indexing crawler. Meta says allowing it helps Meta AI cite and link your content in its responses.
What happens if you block it
Meta AI may stop citing and linking your content in its answers.
Our take
Allow it if you want to be cited in Meta AI.
robots.txt
To block it:
User-agent: meta-webindexer Disallow: /
To allow it explicitly (useful when a wider rule blocks other bots):
User-agent: meta-webindexer Allow: /
The operator's own words
Allowing Meta-WebIndexer in your robots.txt file helps us cite and link to your content in Meta AI's responses.Meta documentation, checked 2026-10-31
robots.txt allows it, but is it really getting through?
A firewall or CDN rule can reject meta-webindexer even when robots.txt welcomes it, and nothing in your CMS shows it. Our check sends a live request as meta-webindexer alongside a normal browser and a Googlebot control, so a bot-verification rule isn't mistaken for a block.
Other Meta agents
Meta's crawler for use cases such as training foundation AI models or improving products by indexing content directly.
Fetches requested by Meta AI users, for example when a user shares a link.
Can meta-webindexer reach your site?
Free, no sign-up. We request your homepage as 13 different clients and show what each one gets.