AI crawler changelog

Pages last fingerprinted Nov 4, 2026

When an AI company adds a crawler, renames one or changes what robots.txt controls, your rules can go stale without you touching them. We fingerprint each operator's crawler page, compare it with the live page, and log a change here only after reading the source.

1. Nov 4, 2026
   Directory update

   Added three agents already listed in their operators' documentation: OpenAI's ad review bot, Google's Vertex AI Agents crawler and Amazon's user-triggered fetcher for Alexa. None of them count toward the grade.

   OAI-AdsBot Google-CloudVertexBot Amzn-User

2. Nov 4, 2026
   Directory update

   Added crawlers from Mistral AI (all three of its agents), You.com, Yandex and Naver. We analyze robots.txt for each; none of them count toward the grade yet.

   MistralAI-User MistralAI-Index MistralAI-Training YouBot YandexBot Yeti

3. Nov 2, 2026
   Directory update

   Added China search engines and Kimi: Huawei PetalBot, Sogou, 360, Shenma (Yisou) and Moonshot's three Kimi agents. Shown in their own group, not graded.

   PetalBot Sogou web spider 360Spider YisouSpider Kimi-SearchBot Kimi-User KimiBot

4. Oct 31, 2026
   Directory update

   Directory launched with 22 crawlers from OpenAI, Anthropic, Google, Apple, Perplexity, Microsoft, Meta, Amazon, DuckDuckGo, Common Crawl, ByteDance and Baidu, each checked against the operator's documentation.

Pages we compare

21 official pages covering 38 crawlers. A change on a page only becomes an entry here after we read it; layout and wording edits that don't affect crawlers are left out.