adidxbot
Bing's ad-landing-page quality crawler.
|
Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +99
|
Redirected
|
—
|
Amzn-SearchBot
Feeds Amazon, Alexa and Rufus search.
|
Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +99
|
HTTP 403
|
the server (HTTP 403)
|
Applebot
The device, OS and WebKit fields vary between Apple's crawlers.
|
Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +99
|
HTTP 403
|
the server (HTTP 403)
|
Applebot-Extended
A robots.txt token with no distinct user-agent: it governs whether Apple Intelligence may train on content Applebot already fetched.
|
Not allowed
by User-agent: applebot-extended
|
Not probed
|
its own robots.txt rule (User-agent: applebot-extended)
|
AspiegelBot
alias — same bot as another row
The legacy name, superseded by PetalBot.
|
Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +99
|
HTTP 403
|
the server (HTTP 403)
|
baidu
alias — same bot as another row
The same bot as Baiduspider, not a separate name or string.
|
Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +99
|
Not probed
|
—
|
|
Baiduspider
|
Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +99
|
HTTP 403
|
the server (HTTP 403)
|
|
Bingbot
|
Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +99
|
Redirected
|
—
|
|
BingPreview
|
Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +99
|
HTTP 403
|
the server (HTTP 403)
|
BingVideoPreview
Bing confirms this bot exists but publishes no literal user-agent string, so only its robots.txt token is checked.
|
Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +99
|
Not probed
|
—
|
|
DuckDuckBot
|
Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +99
|
Redirected
|
—
|
Googlebot
The mobile variant wraps a Chrome/Android string around the same (compatible; Googlebot/2.1; ...) suffix.
|
Not allowed
by User-agent: googlebot
|
Captcha
|
its own robots.txt rule (User-agent: googlebot)
|
|
MicrosoftPreview
|
Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +99
|
HTTP 403
|
the server (HTTP 403)
|
MojeekBot
The version number increments over time (0.2 to 0.11+ seen).
|
Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +99
|
HTTP 403
|
the server (HTTP 403)
|
PetalBot
Huawei Petal Search. The mobile variant prefixes Android and AppleWebKit fields.
|
Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +99
|
HTTP 403
|
the server (HTTP 403)
|
|
SeznamBot
|
Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +99
|
HTTP 403
|
the server (HTTP 403)
|
|
SeznamHomepageCrawler
|
Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +99
|
HTTP 403
|
the server (HTTP 403)
|
|
Slurp
|
Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +99
|
HTTP 403
|
the server (HTTP 403)
|
|
Teoma
|
Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +99
|
HTTP 403
|
the server (HTTP 403)
|
Yahoo-Blogs
Legacy, listed only by unofficial aggregators.
|
Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +99
|
Not probed
|
—
|
Yahoo-FeedSeeker
Historical form, not verified as current.
|
Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +99
|
HTTP 403
|
the server (HTTP 403)
|
Yahoo-MMCrawler
Historical form, not verified as current.
|
Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +99
|
HTTP 403
|
the server (HTTP 403)
|
YahooSeeker
Legacy: Yahoo Search is Bing-powered now and no current documentation publishes a string.
|
Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +99
|
Not probed
|
—
|
Yandex
alias — same bot as another row
A robots.txt control token matching any Yandex* bot, not a distinct request user-agent.
|
Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +99
|
Not probed
|
—
|
YandexAdditional
Mostly a robots.txt control token; this string comes from secondary sources, not from Yandex's own documentation.
|
Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +99
|
HTTP 403
|
the server (HTTP 403)
|
YandexAdditionalBot
alias — same bot as another row
Same bot and same token as YandexAdditional.
|
Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +99
|
Not probed
|
—
|
|
YandexBot
|
Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +99
|
HTTP 403
|
the server (HTTP 403)
|