No single operator · Other
wget
- Operator
- No single operator
- Purpose
- Other (Plumb's classification)
- Verification
- None
- Follows robots.txt
- No robots.txt token
01Identify and verify
How it shows up in your logs
User-agent contains
Wget/
Matching is case-insensitive. Anyone can send this string, so a match alone says only what the request claims to be.
Is it really wget?
The operator publishes no ranges or keys, so any request using this name is unverified.
02Publishers
How major publishers treat it
This agent has no robots.txt token, so the leaderboard cannot grade it.
03robots.txt
No robots.txt rule applies
A general HTTP client, not one company's crawler. It has no robots.txt token and reads robots.txt only if whoever runs it adds that.
04Same operator
No single operator's other agents
- python-requestsOther
- Go-http-clientOther
- curlOther
- axiosOther
- node-fetchOther
- okhttpOther
- ScrapyOther
- HeadlessChromeOther
- libwww-perlOther
- UptimeRobotOther
- jscrawlerOther
- ips-agentOther