Crawlspace · Model training
Crawlspace
- Operator
- Crawlspace
- Purpose
- Model training (Plumb's classification)
- Verification
- None
- Follows robots.txt
- Not documented
01Identify and verify
How it shows up in your logs
User-agent contains
Crawlspace
Matching is case-insensitive. Anyone can send this string, so a match alone says only what the request claims to be.
Is it really Crawlspace?
The operator publishes no ranges or keys, so any request using this name is unverified.
02Publishers
How major publishers treat it
Its robots.txt token is not documented, so the leaderboard cannot grade it.
03robots.txt
robots.txt token not documented
Crawlspace does not document a robots.txt token for Crawlspace, and the leaderboard does not grade one, so Plumb offers no rule to paste. A guessed name would match nothing and block nothing. To stop it, block its user agent at your CDN or firewall.