Bytespider

ByteDance's crawler, documented as the user agent for Toutiao search.

What happens if you block it

Your site may drop out of Toutiao search. ByteDance doesn't document whether blocking it stops AI training use.

Our take

Block it if you don't need Toutiao traffic. Its training use is undocumented.

robots.txt

To block it:

User-agent: Bytespider
Disallow: /

To allow it explicitly (useful when a wider rule blocks other bots):

User-agent: Bytespider
Allow: /

The operator's own words

头条搜索的爬虫UA为“Bytespider”ByteDance documentation, checked 2026-10-31

robots.txt allows it, but is it really getting through?

A firewall or CDN rule can reject Bytespider even when robots.txt welcomes it, and nothing in your CMS shows it. Our check sends a live request as Bytespider alongside a normal browser and a Googlebot control, so a bot-verification rule isn't mistaken for a block.