ByteDance · Model training
Bytespider
- Operator
- ByteDance
- Purpose
- Model training (as the leaderboard grades it)
- Verification
- None
- Follows robots.txt
- Not documented
01Identify and verify
How it shows up in your logs
User-agent contains
Bytespider
Matching is case-insensitive. Anyone can send this string, so a match alone says only what the request claims to be.
Is it really Bytespider?
The operator publishes no ranges or keys, so any request using this name is unverified.
Bytespider publishes no ranges.
02Publishers
How major publishers treat it
56%of 319 publishers on the leaderboard block
Bytespider outright (180). 97 more close some sections. 177 name it explicitly in their robots.txt.03robots.txt
A rule you can paste
The group names Bytespider exactly. A group for a different token from the same company does not cover it.
ByteDance does not document whether this agent follows robots.txt. Plumb can tell you whether it does on your site.
# Bytespider (ByteDance) User-agent: Bytespider Disallow: /
04Same operator
ByteDance's other agents
- TikTokSpiderModel training