You are exploring AkOOre's demo website — read-only, nothing you change is saved.

The AI assistants and the other robots — originaluniqueart.com — generated on Nov. 22, 2026, 11:12 p.m. by AkOOre

OverviewCrawler access · originaluniqueart.com · 16 Nov 2026

Your server turned away every test request, a plain browser's included, so the live test cannot say which bots it lets in: robots.txt alone answers below.

Your robots.txt lets 127 of 157 bots in, and your server served 28 of the 126 it could be asked. The access score is 49/100, with 6 critical checks and 3 to improve. 108 bots get opposite answers from your robots.txt and your server. A plain browser was refused too, so these refusals are not about crawlers.

38%
AI assistants: group score
72%
Social Bots: group score
0 / 21
SEO Tools: bots served
1 / 46
Scrapers: bots served
0 / 4
Cloud Services: bots served
3 / 5
Other Agents: bots served

In this report

6 sections · 157 bots · 8 groups · 39 checks
originaluniqueart.com · 16 Nov 2026

Section A · Scored group

AI assistants

Critical

14 of 33 bots read your site.

Measured Nov. 16, 2026, 2:03 a.m.

0/3 scored points pass. Weakest point: The bots that decide the outcome (12%).

Your robots.txt lets 15 of 33 in. Your server says no to 10.

33 bots knock AI assistants

Gate 1 Your robots.txt

Allowed with exceptionsNot allowed

Your file names and blocks ChatGPT Operator, ChatGPT-User, ChatGPT-User/2.0, Claude-SearchBot, Claude-User, Claude-Web, ClaudeBot, GPTBot and 10 more. 15 of them are allowed with exceptions — the closed paths are housekeeping only, no known content page is among them.

Gate 2 Your server, live

ServedRefusedNot probed

14 Read your site ChatGPT-UserClaude-CodeClaude-SearchBotClaude-UserClaude-WebClaudeBotGPTBotMistralAI-User/1.0OAI-SearchBotPerplexity-User/1.0 + 4 more
10 Turned away by the server AI2Bot HTTP 403Ai2Bot-Dolma HTTP 403DeepSeekBot HTTP 403DigitalOceanGenAICrawler HTTP 403DuckAssistBot HTTP 403LINER Bot HTTP 403PanguBot HTTP 403SBIntuitionsBot HTTP 403cohere-training-data-crawler HTTP 403img2dataset HTTP 403
9 Judged on the file alone ChatGPT OperatorChatGPT-User/2.0Google-ExtendedGrokGrokBotMistralAI-UserMyCentralAIScraperBotPerplexity-Userquillbot.com

Their operator publishes no user-agent string, so no honest knock is possible.

What the measurement means, point by point

Critical Server-robots-instructions — what robots.txt permits 45%

CheckedThe first thing every well-behaved bot reads. A Disallow rule on the homepage stops it before it ever requests a page, so nothing further down can compensate. A group that closes some paths but not the homepage is 'allowed with exceptions' — it counts as allowed, and the exceptions are tested against your known content pages below. A bot no rule names is allowed — silence is permission — and shown as 'not indicated'.

Found18 of 33 not allowed: ChatGPT Operator, ChatGPT-User, ChatGPT-User/2.0, Claude-SearchBot, Claude-User, Claude-Web, ClaudeBot, GPTBot and 10 more; 15 of them are allowed with exceptions — the closed paths are housekeeping only, no known content page is among them.

Do thisYour robots.txt closes the door to ChatGPT Operator, ChatGPT-User, ChatGPT-User/2.0, Claude-SearchBot, Claude-User, Claude-Web, ClaudeBot, GPTBot and 10 more. Open the file and look for the User-agent group named in the table below: unless the block is deliberate, delete its Disallow: / line. This is a one-line change with an outsized effect.

Fix soon Server-crawl-live-test — the live HEAD request 58%

CheckedWhat your server really answers to one HEAD request wearing that bot's name. A firewall, a CDN rule or a bot-protection product can refuse a crawler your robots.txt welcomes — and robots.txt alone can never reveal it.

Found10 forbidden: AI2Bot, Ai2Bot-Dolma, DeepSeekBot, DigitalOceanGenAICrawler, DuckAssistBot, LINER Bot, PanguBot, SBIntuitionsBot and 2 more

Do thisRead this one before acting: an ordinary desktop browser was refused too, so your server is not singling out crawlers — it is refusing our test requests wholesale (a rate limit, a geo or IP rule, or a challenge page). Check what your firewall does with unfamiliar visitors before changing any bot rule.

Worth knowing Indexing directives in the headers

CheckedThe X-Robots-Tag response header of the answers your server actually served. A noindex here lets the bot in and then tells it to throw the page away — the most expensive way to be invisible, because it looks fine in every other check. Only served answers are read: a wall's or an error page's header is not yours.

Foundno X-Robots-Tag header in any served answer

Worth knowing Do the two tests agree?

CheckedWhether the server-robots-instructions and the server-crawl-live-test tell the same story. They often do not: robots.txt is a published promise, the live answer is what actually happens, and only the comparison shows which one to believe.

Found10 bot(s) invited by robots.txt but refused by the server — the same ones the live test above names; 13 bot(s) blocked in robots.txt but served anyway: ChatGPT-User, Claude-SearchBot, Claude-User, Claude-Web, ClaudeBot, GPTBot, MistralAI-User/1.0, OAI-SearchBot and 5 more

Worth knowingA contradiction is worth resolving whichever way you meant it. Invited-but-refused: your firewall overrules your robots.txt — align them, or accept that the file is advertising a welcome the server does not honour. Blocked-but-served: your robots.txt line is a polite request only; a bot that ignores it gets the page, so add a server rule if the block really matters.

Critical The bots that decide the outcome 12%

CheckedNot every bot in this group is worth the same. These are the ones whose verdict changes what your business gets: the indexes people actually search and the assistants they actually ask.

FoundChatGPT-User is stopped by its own robots.txt rule (User-agent: chatgpt-user); Claude-SearchBot is stopped by its own robots.txt rule (User-agent: claude-searchbot); ClaudeBot is stopped by its own robots.txt rule (User-agent: claudebot); GPTBot is stopped by its own robots.txt rule (User-agent: gptbot); Google-Extended is stopped by its own robots.txt rule (User-agent: google-extended); OAI-SearchBot is stopped by its own robots.txt rule (User-agent: oai-searchbot); PerplexityBot is stopped by its own robots.txt rule (User-agent: perplexitybot); your server refused every test request, a plain browser's included, so its answer to DuckAssistBot is not counted here

Do thisFix these before anything else in this report: each one is a whole audience that cannot see the site today. The table below names the exact rule or answer responsible for each.

Worth knowing Judged on robots.txt alone

CheckedSome operators publish a robots.txt token but never a literal user-agent string. Those bots cannot be impersonated honestly, so they are judged on your rules only — no live request was made in their name.

Found9 of 33: ChatGPT Operator, ChatGPT-User/2.0, Google-Extended, Grok, GrokBot, MistralAI-User, MyCentralAIScraperBot, Perplexity-User and 1 more

Worth knowingNothing to do. Their robots.txt verdict is still exact — it is read from your own file — only the server-side half is unknown.

The AI assistants group, bot by bot

  1. ✗

    AI2Bot

    robots.txt: Allowed with exceptions · your server: HTTP 403

  2. ✗

    Ai2Bot-Dolma

    robots.txt: Allowed with exceptions · your server: HTTP 403

  3. ✓

    anthropic-ai

    robots.txt: Not allowed · your server: Redirected · its own robots.txt rule (User-agent: anthropic-ai)

  4. !

    ChatGPT Operator

    robots.txt: Not allowed · your server: Not probed · its own robots.txt rule (User-agent: chatgpt-user)

  5. ✓

    ChatGPT-User

    robots.txt: Not allowed · your server: Redirected · its own robots.txt rule (User-agent: chatgpt-user)

  6. !

    ChatGPT-User/2.0

    robots.txt: Not allowed · your server: Not probed · its own robots.txt rule (User-agent: chatgpt-user)

  7. ✓

    Claude-Code

    robots.txt: Allowed with exceptions · your server: Redirected

  8. ✓

    Claude-SearchBot

    robots.txt: Not allowed · your server: Redirected · its own robots.txt rule (User-agent: claude-searchbot)

  9. ✓

    Claude-User

    robots.txt: Not allowed · your server: Redirected · its own robots.txt rule (User-agent: claude-user)

  10. ✓

    Claude-Web

    robots.txt: Not allowed · your server: Redirected · its own robots.txt rule (User-agent: claude-web)

  11. ✓

    ClaudeBot

    robots.txt: Not allowed · your server: Redirected · its own robots.txt rule (User-agent: claudebot)

  12. ✓

    cohere-ai

    robots.txt: Not allowed · your server: Redirected · its own robots.txt rule (User-agent: cohere-ai)

  13. ✗

    cohere-training-data-crawler

    robots.txt: Allowed with exceptions · your server: HTTP 403

  14. ✗

    DeepSeekBot

    robots.txt: Allowed with exceptions · your server: HTTP 403

  15. ✗

    DigitalOceanGenAICrawler

    robots.txt: Allowed with exceptions · your server: HTTP 403

  16. ✗

    DuckAssistBot

    robots.txt: Allowed with exceptions · your server: HTTP 403

  17. !

    Google-Extended

    robots.txt: Not allowed · your server: Not probed · its own robots.txt rule (User-agent: google-extended)

  18. ✓

    GPTBot

    robots.txt: Not allowed · your server: Redirected · its own robots.txt rule (User-agent: gptbot)

  19. !

    Grok

    robots.txt: Allowed with exceptions · your server: Not probed

  20. !

    GrokBot

    robots.txt: Allowed with exceptions · your server: Not probed

  21. ✗

    img2dataset

    robots.txt: Allowed with exceptions · your server: HTTP 403

  22. ✗

    LINER Bot

    robots.txt: Allowed with exceptions · your server: HTTP 403

  23. !

    MistralAI-User

    robots.txt: Not allowed · your server: Not probed · its own robots.txt rule (User-agent: mistralai-user)

  24. ✓

    MistralAI-User/1.0

    robots.txt: Not allowed · your server: Redirected · its own robots.txt rule (User-agent: mistralai-user)

  25. !

    MyCentralAIScraperBot

    robots.txt: Allowed with exceptions · your server: Not probed

  26. ✓

    OAI-SearchBot

    robots.txt: Not allowed · your server: Redirected · its own robots.txt rule (User-agent: oai-searchbot)

  27. ✗

    PanguBot

    robots.txt: Allowed with exceptions · your server: HTTP 403

  28. !

    Perplexity-User

    robots.txt: Not allowed · your server: Not probed · its own robots.txt rule (User-agent: perplexity-user)

  29. ✓

    Perplexity-User/1.0

    robots.txt: Not allowed · your server: Redirected · its own robots.txt rule (User-agent: perplexity-user)

  30. ✓

    PerplexityBot

    robots.txt: Not allowed · your server: Redirected · its own robots.txt rule (User-agent: perplexitybot)

  31. !

    quillbot.com

    robots.txt: Allowed with exceptions · your server: Not probed

  32. ✗

    SBIntuitionsBot

    robots.txt: Allowed with exceptions · your server: HTTP 403

  33. ✓

    YouBot

    robots.txt: Not allowed · your server: Redirected · its own robots.txt rule (User-agent: youbot)

See the roster table
Every bot of the AI assistants group, with its two verdicts
Bot Gate 1 · your robots.txt Gate 2 · your server, live What stopped it
AI2Bot
A bare token.
Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +99
HTTP 403 the server (HTTP 403)
Ai2Bot-Dolma Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +99
HTTP 403 the server (HTTP 403)
anthropic-ai
alias — same bot as another row
A deprecated legacy token, superseded by ClaudeBot.
Not allowed
by User-agent: anthropic-ai
Redirected its own robots.txt rule (User-agent: anthropic-ai)
ChatGPT Operator
alias — same bot as another row
Folded into ChatGPT agent in 2025. It sends no distinctive user-agent -- it uses an ordinary Chrome desktop string and self-identifies with a Signature-Agent: https://chatgpt.com header, so it cannot be impersonated by user-agent alone.
Not allowed
by User-agent: chatgpt-user
Not probed its own robots.txt rule (User-agent: chatgpt-user)
ChatGPT-User
OpenAI's live fetch when a ChatGPT user asks about your page.
Not allowed
by User-agent: chatgpt-user
Redirected its own robots.txt rule (User-agent: chatgpt-user)
ChatGPT-User/2.0
alias — same bot as another row
OpenAI's published version is 1.0; there is no evidence of a 2.0, so only the token is checked.
Not allowed
by User-agent: chatgpt-user
Not probed its own robots.txt rule (User-agent: chatgpt-user)
Claude-Code
alias — same bot as another row
Not a separately documented crawler: the Claude Code CLI's own fetches ride the Claude-User pattern.
Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +99
Redirected —
Claude-SearchBot Not allowed
by User-agent: claude-searchbot
Redirected its own robots.txt rule (User-agent: claude-searchbot)
Claude-User Not allowed
by User-agent: claude-user
Redirected its own robots.txt rule (User-agent: claude-user)
Claude-Web
alias — same bot as another row
Deprecated, replaced by ClaudeBot / Claude-User / Claude-SearchBot. The string is historical.
Not allowed
by User-agent: claude-web
Redirected its own robots.txt rule (User-agent: claude-web)
ClaudeBot Not allowed
by User-agent: claudebot
Redirected its own robots.txt rule (User-agent: claudebot)
cohere-ai
A bare token; no Mozilla-wrapped form is documented.
Not allowed
by User-agent: cohere-ai
Redirected its own robots.txt rule (User-agent: cohere-ai)
cohere-training-data-crawler Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +99
HTTP 403 the server (HTTP 403)
DeepSeekBot Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +99
HTTP 403 the server (HTTP 403)
DigitalOceanGenAICrawler
Names conflict: DigitalOcean's current docs say DigitalOceanGradientAICrawler/1.0, a widely cited third-party report says DigitalOceanGenAICrawler/1.0.
Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +99
HTTP 403 the server (HTTP 403)
DuckAssistBot Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +99
HTTP 403 the server (HTTP 403)
Google-Extended
A robots.txt token only: the crawl happens under the standard Googlebot or GoogleOther user-agent. It governs whether your pages may ground Gemini answers and AI Overviews.
Not allowed
by User-agent: google-extended
Not probed its own robots.txt rule (User-agent: google-extended)
GPTBot Not allowed
by User-agent: gptbot
Redirected its own robots.txt rule (User-agent: gptbot)
Grok
xAI publishes no crawler documentation page; third-party trackers disagree (Grok, xAI-Grok/1.0, GrokBot/1.0), so no string is sent.
Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +99
Not probed —
GrokBot
alias — same bot as another row
Same caveat as Grok: no official documentation.
Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +99
Not probed —
img2dataset
An open-source tool run by many independent operators.
Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +99
HTTP 403 the server (HTTP 403)
LINER Bot Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +99
HTTP 403 the server (HTTP 403)
MistralAI-User
alias — same bot as another row
1.0 is the only published version.
Not allowed
by User-agent: mistralai-user
Not probed its own robots.txt rule (User-agent: mistralai-user)
MistralAI-User/1.0 Not allowed
by User-agent: mistralai-user
Redirected its own robots.txt rule (User-agent: mistralai-user)
MyCentralAIScraperBot
The token circulates in blocklists; the operator and the user-agent string are undocumented.
Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +99
Not probed —
OAI-SearchBot
Builds the index behind ChatGPT search results.
Not allowed
by User-agent: oai-searchbot
Redirected its own robots.txt rule (User-agent: oai-searchbot)
PanguBot
Huawei's Pangu LLM crawler.
Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +99
HTTP 403 the server (HTTP 403)
Perplexity-User
alias — same bot as another row
Same bot as Perplexity-User/1.0.
Not allowed
by User-agent: perplexity-user
Not probed its own robots.txt rule (User-agent: perplexity-user)
Perplexity-User/1.0 Not allowed
by User-agent: perplexity-user
Redirected its own robots.txt rule (User-agent: perplexity-user)
PerplexityBot Not allowed
by User-agent: perplexitybot
Redirected its own robots.txt rule (User-agent: perplexitybot)
quillbot.com
The robots.txt token is confirmed; no literal user-agent string is published.
Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +99
Not probed —
SBIntuitionsBot Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +99
HTTP 403 the server (HTTP 403)
YouBot Not allowed
by User-agent: youbot
Redirected its own robots.txt rule (User-agent: youbot)

Section B · Scored group

Social Bots

Fix soon

7 of 11 bots read your site.

Measured Nov. 16, 2026, 2:03 a.m.

1/3 scored points pass. Weakest point: Server-crawl-live-test — the live HEAD request (64%).

Your robots.txt lets 8 of 11 in. Your server says no to 4.

11 bots knock Social Bots

Gate 1 Your robots.txt

Allowed with exceptionsNot allowed

Your file names and blocks FacebookBot, LinkedInBot, Meta-ExternalAgent. 8 of them are allowed with exceptions — the closed paths are housekeeping only, no known content page is among them.

Gate 2 Your server, live

ServedRefused

7 Read your site FacebookBotLinkedInBotMeta-ExternalAgentPinterestbotSlackbotTwitterbotfacebookexternalhit
4 Turned away by the server Meta-ExternalAds HTTP 403Meta-ExternalFetcher HTTP 403Meta-WebIndexer HTTP 403Quora-Bot HTTP 403

What the measurement means, point by point

Fix soon Server-robots-instructions — what robots.txt permits 73%

CheckedThe first thing every well-behaved bot reads. A Disallow rule on the homepage stops it before it ever requests a page, so nothing further down can compensate. A group that closes some paths but not the homepage is 'allowed with exceptions' — it counts as allowed, and the exceptions are tested against your known content pages below. A bot no rule names is allowed — silence is permission — and shown as 'not indicated'.

Found3 of 11 not allowed: FacebookBot, LinkedInBot, Meta-ExternalAgent; 8 of them are allowed with exceptions — the closed paths are housekeeping only, no known content page is among them.

Do thisYour robots.txt closes the door to FacebookBot, LinkedInBot, Meta-ExternalAgent. Open the file and look for the User-agent group named in the table below: unless the block is deliberate, delete its Disallow: / line. This is a one-line change with an outsized effect.

Fix soon Server-crawl-live-test — the live HEAD request 64%

CheckedWhat your server really answers to one HEAD request wearing that bot's name. A firewall, a CDN rule or a bot-protection product can refuse a crawler your robots.txt welcomes — and robots.txt alone can never reveal it.

Found4 forbidden: Meta-ExternalAds, Meta-ExternalFetcher, Meta-WebIndexer, Quora-Bot

Do thisRead this one before acting: an ordinary desktop browser was refused too, so your server is not singling out crawlers — it is refusing our test requests wholesale (a rate limit, a geo or IP rule, or a challenge page). Check what your firewall does with unfamiliar visitors before changing any bot rule.

Worth knowing Indexing directives in the headers

CheckedThe X-Robots-Tag response header of the answers your server actually served. A noindex here lets the bot in and then tells it to throw the page away — the most expensive way to be invisible, because it looks fine in every other check. Only served answers are read: a wall's or an error page's header is not yours.

Foundno X-Robots-Tag header in any served answer

Worth knowing Do the two tests agree?

CheckedWhether the server-robots-instructions and the server-crawl-live-test tell the same story. They often do not: robots.txt is a published promise, the live answer is what actually happens, and only the comparison shows which one to believe.

Found4 bot(s) invited by robots.txt but refused by the server — the same ones the live test above names; 3 bot(s) blocked in robots.txt but served anyway: FacebookBot, LinkedInBot, Meta-ExternalAgent

Worth knowingA contradiction is worth resolving whichever way you meant it. Invited-but-refused: your firewall overrules your robots.txt — align them, or accept that the file is advertising a welcome the server does not honour. Blocked-but-served: your robots.txt line is a polite request only; a bot that ignores it gets the page, so add a server rule if the block really matters.

Good The bots that decide the outcome 80%

CheckedNot every bot in this group is worth the same. These are the ones whose verdict changes what your business gets: the indexes people actually search and the assistants they actually ask.

FoundLinkedInBot is stopped by its own robots.txt rule (User-agent: linkedinbot)

The Social Bots group, bot by bot

  1. ✓

    FacebookBot

    robots.txt: Not allowed · your server: Redirected · its own robots.txt rule (User-agent: facebookbot)

  2. ✓

    facebookexternalhit

    robots.txt: Allowed with exceptions · your server: Redirected

  3. ✓

    LinkedInBot

    robots.txt: Not allowed · your server: Redirected · its own robots.txt rule (User-agent: linkedinbot)

  4. ✗

    Meta-ExternalAds

    robots.txt: Allowed with exceptions · your server: HTTP 403

  5. ✓

    Meta-ExternalAgent

    robots.txt: Not allowed · your server: Redirected · its own robots.txt rule (User-agent: meta-externalagent)

  6. ✗

    Meta-ExternalFetcher

    robots.txt: Allowed with exceptions · your server: HTTP 403

  7. ✗

    Meta-WebIndexer

    robots.txt: Allowed with exceptions · your server: HTTP 403

  8. ✓

    Pinterestbot

    robots.txt: Allowed with exceptions · your server: Redirected

  9. ✗

    Quora-Bot

    robots.txt: Allowed with exceptions · your server: HTTP 403

  10. ✓

    Slackbot

    robots.txt: Allowed with exceptions · your server: Redirected

  11. ✓

    Twitterbot

    robots.txt: Allowed with exceptions · your server: Redirected

See the roster table
Every bot of the Social Bots group, with its two verdicts
Bot Gate 1 · your robots.txt Gate 2 · your server, live What stopped it
FacebookBot
Meta's general research/product crawler, distinct from facebookexternalhit.
Not allowed
by User-agent: facebookbot
Redirected its own robots.txt rule (User-agent: facebookbot)
facebookexternalhit
The link-preview fetcher; also seen bare as facebookexternalhit/1.1.
Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +99
Redirected —
LinkedInBot
The HttpClient version field varies.
Not allowed
by User-agent: linkedinbot
Redirected its own robots.txt rule (User-agent: linkedinbot)
Meta-ExternalAds
Ad-landing-page review.
Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +99
HTTP 403 the server (HTTP 403)
Meta-ExternalAgent
AI training plus search indexing.
Not allowed
by User-agent: meta-externalagent
Redirected its own robots.txt rule (User-agent: meta-externalagent)
Meta-ExternalFetcher
On-demand fetch for Meta AI features.
Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +99
HTTP 403 the server (HTTP 403)
Meta-WebIndexer
Meta's newer (2025) general web-indexing crawler.
Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +99
HTTP 403 the server (HTTP 403)
Pinterestbot
A lighter fetcher is also seen: Pinterest/0.2 (+https://www.pinterest.com/bot.html).
Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +99
Redirected —
Quora-Bot Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +99
HTTP 403 the server (HTTP 403)
Slackbot
Other Slack integrations use Slackbot 1.0 (+https://api.slack.com/robots).
Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +99
Redirected —
Twitterbot Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +99
Redirected —

Section C · Measured group

SEO Tools

0 of 21 bots read your site.

Measured Nov. 16, 2026, 2:03 a.m.

Your robots.txt says yes to all 21. Your server says no to 20.

21 bots knock SEO Tools

Gate 1 Your robots.txt

All of them ride on your catch-all group (User-agent: *) — no rule in your file names one of them individually. 21 of them are allowed with exceptions — the closed paths are housekeeping only, no known content page is among them.

Gate 2 Your server, live

RefusedNot probed

20 Turned away by the server AhrefsBot HTTP 403AudigentAdBot HTTP 403AwarioRssBot HTTP 403AwarioSmartBot HTTP 403Brandwatch HTTP 403Chrome-Lighthouse HTTP 403DataForSeoBot HTTP 403DotBot HTTP 403Google Page Speed Insights HTTP 403Jetslide HTTP 403 + 10 more
1 Judged on the file alone NeticleBot

Their operator publishes no user-agent string, so no honest knock is possible.

What the measurement means, point by point

Worth knowing Server-robots-instructions — what robots.txt permits

CheckedThe first thing every well-behaved bot reads. A Disallow rule on the homepage stops it before it ever requests a page, so nothing further down can compensate. A group that closes some paths but not the homepage is 'allowed with exceptions' — it counts as allowed, and the exceptions are tested against your known content pages below. A bot no rule names is allowed — silence is permission — and shown as 'not indicated'.

Foundall 21 are allowed by robots.txt; 21 of them are allowed with exceptions — the closed paths are housekeeping only, no known content page is among them.

Worth knowing Server-crawl-live-test — the live HEAD request

CheckedWhat your server really answers to one HEAD request wearing that bot's name. A firewall, a CDN rule or a bot-protection product can refuse a crawler your robots.txt welcomes — and robots.txt alone can never reveal it.

Found20 forbidden: AhrefsBot, AudigentAdBot, AwarioRssBot, AwarioSmartBot, Brandwatch, Chrome-Lighthouse, DataForSeoBot, DotBot and 12 more

Worth knowingRead this one before acting: an ordinary desktop browser was refused too, so your server is not singling out crawlers — it is refusing our test requests wholesale (a rate limit, a geo or IP rule, or a challenge page). Check what your firewall does with unfamiliar visitors before changing any bot rule.

Worth knowing Do the two tests agree?

CheckedWhether the server-robots-instructions and the server-crawl-live-test tell the same story. They often do not: robots.txt is a published promise, the live answer is what actually happens, and only the comparison shows which one to believe.

Found20 bot(s) invited by robots.txt but refused by the server — the same ones the live test above names

Worth knowingA contradiction is worth resolving whichever way you meant it. Invited-but-refused: your firewall overrules your robots.txt — align them, or accept that the file is advertising a welcome the server does not honour. Blocked-but-served: your robots.txt line is a polite request only; a bot that ignores it gets the page, so add a server rule if the block really matters.

Worth knowing Judged on robots.txt alone

CheckedSome operators publish a robots.txt token but never a literal user-agent string. Those bots cannot be impersonated honestly, so they are judged on your rules only — no live request was made in their name.

Found1 of 21: NeticleBot

Worth knowingNothing to do. Their robots.txt verdict is still exact — it is read from your own file — only the server-side half is unknown.

The SEO Tools group, bot by bot

  1. ✗

    AhrefsBot

    robots.txt: Allowed with exceptions · your server: HTTP 403

  2. ✗

    AudigentAdBot

    robots.txt: Allowed with exceptions · your server: HTTP 403

  3. ✗

    AwarioRssBot

    robots.txt: Allowed with exceptions · your server: HTTP 403

  4. ✗

    AwarioSmartBot

    robots.txt: Allowed with exceptions · your server: HTTP 403

  5. ✗

    Brandwatch

    robots.txt: Allowed with exceptions · your server: HTTP 403

  6. ✗

    Chrome-Lighthouse

    robots.txt: Allowed with exceptions · your server: HTTP 403

  7. ✗

    DataForSeoBot

    robots.txt: Allowed with exceptions · your server: HTTP 403

  8. ✗

    DotBot

    robots.txt: Allowed with exceptions · your server: HTTP 403

  9. ✗

    Google Page Speed Insights

    robots.txt: Allowed with exceptions · your server: HTTP 403

  10. ✗

    Jetslide

    robots.txt: Allowed with exceptions · your server: HTTP 403

  11. ✗

    Meltwater

    robots.txt: Allowed with exceptions · your server: HTTP 403

  12. ✗

    MJ12bot

    robots.txt: Allowed with exceptions · your server: HTTP 403

  13. !

    NeticleBot

    robots.txt: Allowed with exceptions · your server: Not probed

  14. ✗

    Netvibes

    robots.txt: Allowed with exceptions · your server: HTTP 403

  15. ✗

    peer39_crawler

    robots.txt: Allowed with exceptions · your server: HTTP 403

  16. ✗

    Screaming Frog SEO Spider

    robots.txt: Allowed with exceptions · your server: HTTP 403

  17. ✗

    SearchmetricsBot

    robots.txt: Allowed with exceptions · your server: HTTP 403

  18. ✗

    SemrushBot

    robots.txt: Allowed with exceptions · your server: HTTP 403

  19. ✗

    SemrushBot-OCOB

    robots.txt: Allowed with exceptions · your server: HTTP 403

  20. ✗

    SemrushBotSwa

    robots.txt: Allowed with exceptions · your server: HTTP 403

  21. ✗

    Sidetrade Indexer Bot

    robots.txt: Allowed with exceptions · your server: HTTP 403

See the roster table
Every bot of the SEO Tools group, with its two verdicts
Bot Gate 1 · your robots.txt Gate 2 · your server, live What stopped it
AhrefsBot Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +99
HTTP 403 the server (HTTP 403)
AudigentAdBot Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +99
HTTP 403 the server (HTTP 403)
AwarioRssBot Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +99
HTTP 403 the server (HTTP 403)
AwarioSmartBot Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +99
HTTP 403 the server (HTTP 403)
Brandwatch
The vendor's own page says the string varies, so this literal is low confidence.
Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +99
HTTP 403 the server (HTTP 403)
Chrome-Lighthouse Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +99
HTTP 403 the server (HTTP 403)
DataForSeoBot
The crawler behind the data AkOOre's own reports are built on.
Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +99
HTTP 403 the server (HTTP 403)
DotBot
A newer DotBot/2.0; +https://dotbot.com/about/ form is also reported but unconfirmed.
Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +99
HTTP 403 the server (HTTP 403)
Google Page Speed Insights
alias — same bot as another row
The documented legacy form. Recent PSI requests reportedly drop these tokens for a plain Chrome user-agent.
Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +99
HTTP 403 the server (HTTP 403)
Jetslide Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +99
HTTP 403 the server (HTTP 403)
Meltwater Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +99
HTTP 403 the server (HTTP 403)
MJ12bot
Newer v2.0.x builds are also reported, same format.
Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +99
HTTP 403 the server (HTTP 403)
NeticleBot
The vendor has a bot-info page but quotes no exact header.
Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +99
Not probed —
Netvibes Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +99
HTTP 403 the server (HTTP 403)
peer39_crawler
Ad-context classification. The source list names it twice (SEO Tools and Other Agents); it is one bot, listed once.
Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +99
HTTP 403 the server (HTTP 403)
Screaming Frog SEO Spider
The version tracks the installed app and changes every release.
Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +99
HTTP 403 the server (HTTP 403)
SearchmetricsBot Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +99
HTTP 403 the server (HTTP 403)
SemrushBot
Older SemrushBot-BA / -SA / -SI variants also exist.
Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +99
HTTP 403 the server (HTTP 403)
SemrushBot-OCOB
The Content Toolkit crawler.
Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +99
HTTP 403 the server (HTTP 403)
SemrushBotSwa Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +99
HTTP 403 the server (HTTP 403)
Sidetrade Indexer Bot Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +99
HTTP 403 the server (HTTP 403)

Section D · Measured group

Scrapers

1 of 46 bots read your site.

Measured Nov. 16, 2026, 2:03 a.m.

Your robots.txt says yes to all 46. Your server says no to 34.

46 bots knock Scrapers

Gate 1 Your robots.txt

All of them ride on your catch-all group (User-agent: *) — no rule in your file names one of them individually. 46 of them are allowed with exceptions — the closed paths are housekeeping only, no known content page is among them.

Gate 2 Your server, live

ServedCaptcha wallRefusedNot probed

1 Read your site Barkrowler
34 Turned away by the server Arquivo-web-crawler HTTP 403BLEXBot HTTP 403Bytespider HTTP 403Cotoyogi HTTP 403Crawl4AI HTTP 403Diffbot HTTP 403EchoboxBot HTTP 403Factset_spyderbot HTTP 403Firecrawl HTTP 403FriendlyCrawler HTTP 403 + 24 more
11 Judged on the file alone Amazon KendraBravestCrawlspaceEchobot BotJenkersBotPoseidon Research CrawlerSeekrSeekrBothada.newsimediaethics.org + 1 more

Their operator publishes no user-agent string, so no honest knock is possible.

What the measurement means, point by point

Worth knowing Server-robots-instructions — what robots.txt permits

CheckedThe first thing every well-behaved bot reads. A Disallow rule on the homepage stops it before it ever requests a page, so nothing further down can compensate. A group that closes some paths but not the homepage is 'allowed with exceptions' — it counts as allowed, and the exceptions are tested against your known content pages below. A bot no rule names is allowed — silence is permission — and shown as 'not indicated'.

Foundall 46 are allowed by robots.txt; 46 of them are allowed with exceptions — the closed paths are housekeeping only, no known content page is among them.

Worth knowing Server-crawl-live-test — the live HEAD request

CheckedWhat your server really answers to one HEAD request wearing that bot's name. A firewall, a CDN rule or a bot-protection product can refuse a crawler your robots.txt welcomes — and robots.txt alone can never reveal it.

Found34 forbidden: Arquivo-web-crawler, BLEXBot, Bytespider, CCBot, Cotoyogi, Crawl4AI, Diffbot, EchoboxBot and 26 more

Worth knowingRead this one before acting: an ordinary desktop browser was refused too, so your server is not singling out crawlers — it is refusing our test requests wholesale (a rate limit, a geo or IP rule, or a challenge page). Check what your firewall does with unfamiliar visitors before changing any bot rule.

Worth knowing Indexing directives in the headers

CheckedThe X-Robots-Tag response header of the answers your server actually served. A noindex here lets the bot in and then tells it to throw the page away — the most expensive way to be invisible, because it looks fine in every other check. Only served answers are read: a wall's or an error page's header is not yours.

Foundno X-Robots-Tag header in any served answer

Worth knowing Do the two tests agree?

CheckedWhether the server-robots-instructions and the server-crawl-live-test tell the same story. They often do not: robots.txt is a published promise, the live answer is what actually happens, and only the comparison shows which one to believe.

Found34 bot(s) invited by robots.txt but refused by the server — the same ones the live test above names

Worth knowingA contradiction is worth resolving whichever way you meant it. Invited-but-refused: your firewall overrules your robots.txt — align them, or accept that the file is advertising a welcome the server does not honour. Blocked-but-served: your robots.txt line is a polite request only; a bot that ignores it gets the page, so add a server rule if the block really matters.

Worth knowing Judged on robots.txt alone

CheckedSome operators publish a robots.txt token but never a literal user-agent string. Those bots cannot be impersonated honestly, so they are judged on your rules only — no live request was made in their name.

Found11 of 46: Amazon Kendra, Bravest, Crawlspace, Echobot Bot, JenkersBot, Poseidon Research Crawler, Seekr, SeekrBot and 3 more

Worth knowingNothing to do. Their robots.txt verdict is still exact — it is read from your own file — only the server-side half is unknown.

The Scrapers group, bot by bot

  1. !

    Amazon Kendra

    robots.txt: Allowed with exceptions · your server: Not probed

  2. ✗

    Arquivo-web-crawler

    robots.txt: Allowed with exceptions · your server: HTTP 403

  3. ✓

    Barkrowler

    robots.txt: Allowed with exceptions · your server: Redirected

  4. ✗

    BLEXBot

    robots.txt: Allowed with exceptions · your server: HTTP 403

  5. !

    Bravest

    robots.txt: Allowed with exceptions · your server: Not probed

  6. ✗

    Bytespider

    robots.txt: Allowed with exceptions · your server: HTTP 403

  7. ✗

    CCBot

    robots.txt: Allowed with exceptions · your server: Captcha · a bot wall instead of the page — redirected to a “captcha” page

  8. ✗

    coccocbot-web

    robots.txt: Allowed with exceptions · your server: HTTP 403

  9. ✗

    Cotoyogi

    robots.txt: Allowed with exceptions · your server: HTTP 403

  10. ✗

    Crawl4AI

    robots.txt: Allowed with exceptions · your server: HTTP 403

  11. ✗

    crawler4j

    robots.txt: Allowed with exceptions · your server: HTTP 403

  12. !

    Crawlspace

    robots.txt: Allowed with exceptions · your server: Not probed

  13. ✗

    Diffbot

    robots.txt: Allowed with exceptions · your server: HTTP 403

  14. !

    Echobot Bot

    robots.txt: Allowed with exceptions · your server: Not probed

  15. ✗

    EchoboxBot

    robots.txt: Allowed with exceptions · your server: HTTP 403

  16. ✗

    Factset_spyderbot

    robots.txt: Allowed with exceptions · your server: HTTP 403

  17. ✗

    Firecrawl

    robots.txt: Allowed with exceptions · your server: HTTP 403

  18. ✗

    FriendlyCrawler

    robots.txt: Allowed with exceptions · your server: HTTP 403

  19. !

    hada.news

    robots.txt: Allowed with exceptions · your server: Not probed

  20. ✗

    iaskspider

    robots.txt: Allowed with exceptions · your server: HTTP 403

  21. ✗

    ICC-Crawler

    robots.txt: Allowed with exceptions · your server: HTTP 403

  22. ✗

    ImagesiftBot

    robots.txt: Allowed with exceptions · your server: HTTP 403

  23. !

    imediaethics.org

    robots.txt: Allowed with exceptions · your server: Not probed

  24. ✗

    imgproxy

    robots.txt: Allowed with exceptions · your server: HTTP 403

  25. ✗

    ISSCyberRiskCrawler

    robots.txt: Allowed with exceptions · your server: HTTP 403

  26. !

    JenkersBot

    robots.txt: Allowed with exceptions · your server: Not probed

  27. ✗

    Kangaroo Bot

    robots.txt: Allowed with exceptions · your server: HTTP 403

  28. ✗

    magpie-crawler

    robots.txt: Allowed with exceptions · your server: HTTP 403

  29. !

    netEstate Imprint Crawler

    robots.txt: Allowed with exceptions · your server: Not probed

  30. ✗

    news-please

    robots.txt: Allowed with exceptions · your server: HTTP 403

  31. ✗

    NewsNow

    robots.txt: Allowed with exceptions · your server: HTTP 403

  32. ✗

    NovaAct

    robots.txt: Allowed with exceptions · your server: HTTP 403

  33. ✗

    omgili

    robots.txt: Allowed with exceptions · your server: HTTP 403

  34. ✗

    omgilibot

    robots.txt: Allowed with exceptions · your server: HTTP 403

  35. !

    Poseidon Research Crawler

    robots.txt: Allowed with exceptions · your server: Not probed

  36. ✗

    QualifiedBot

    robots.txt: Allowed with exceptions · your server: HTTP 403

  37. ✗

    Scrapy

    robots.txt: Allowed with exceptions · your server: HTTP 403

  38. ✗

    SeekportBot

    robots.txt: Allowed with exceptions · your server: HTTP 403

  39. !

    Seekr

    robots.txt: Allowed with exceptions · your server: Not probed

  40. !

    SeekrBot

    robots.txt: Allowed with exceptions · your server: Not probed

  41. ✗

    TaraGroup Intelligent Bot

    robots.txt: Allowed with exceptions · your server: HTTP 403

  42. ✗

    Timpibot

    robots.txt: Allowed with exceptions · your server: HTTP 403

  43. ✗

    Turnitin

    robots.txt: Allowed with exceptions · your server: HTTP 403

  44. ✗

    VelenPublicWebCrawler

    robots.txt: Allowed with exceptions · your server: HTTP 403

  45. ✗

    Webzio-Extended

    robots.txt: Allowed with exceptions · your server: HTTP 403

  46. ✗

    yacybot

    robots.txt: Allowed with exceptions · your server: HTTP 403

See the roster table
Every bot of the Scrapers group, with its two verdicts
Bot Gate 1 · your robots.txt Gate 2 · your server, live What stopped it
Amazon Kendra
AWS publishes the robots.txt token but no literal HTTP header.
Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +99
Not probed —
Arquivo-web-crawler Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +99
HTTP 403 the server (HTTP 403)
Barkrowler Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +99
Redirected —
BLEXBot Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +99
HTTP 403 the server (HTTP 403)
Bravest
No published user-agent string.
Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +99
Not probed —
Bytespider
ByteDance never published an official documentation page; the Bytespider token is the stable part.
Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +99
HTTP 403 the server (HTTP 403)
CCBot
Common Crawl feeds most open models' training sets.
Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +99
Captcha a bot wall instead of the page — redirected to a “captcha” page
coccocbot-web Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +99
HTTP 403 the server (HTTP 403)
Cotoyogi Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +99
HTTP 403 the server (HTTP 403)
Crawl4AI
A library default the caller can override.
Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +99
HTTP 403 the server (HTTP 403)
crawler4j
A library default, set per deployment by the developer.
Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +99
HTTP 403 the server (HTTP 403)
Crawlspace
No published user-agent string.
Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +99
Not probed —
Diffbot
Version numbers beyond 0.1 are unverified.
Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +99
HTTP 403 the server (HTTP 403)
Echobot Bot
Not to be confused with EchoboxBot above: different company, different bot.
Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +99
Not probed —
EchoboxBot
The hash segment is per-deployment.
Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +99
HTTP 403 the server (HTTP 403)
Factset_spyderbot Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +99
HTTP 403 the server (HTTP 403)
Firecrawl Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +99
HTTP 403 the server (HTTP 403)
FriendlyCrawler Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +99
HTTP 403 the server (HTTP 403)
hada.news
No published user-agent string.
Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +99
Not probed —
iaskspider Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +99
HTTP 403 the server (HTTP 403)
ICC-Crawler Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +99
HTTP 403 the server (HTTP 403)
ImagesiftBot Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +99
HTTP 403 the server (HTTP 403)
imediaethics.org
No published user-agent string.
Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +99
Not probed —
imgproxy
A self-hosted image-proxy tool rather than a crawler; the version tracks its release.
Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +99
HTTP 403 the server (HTTP 403)
ISSCyberRiskCrawler Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +99
HTTP 403 the server (HTTP 403)
JenkersBot
No published user-agent string.
Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +99
Not probed —
Kangaroo Bot
A differently spelled KangarooBot/1.0; +https://kangaroobot.com (no space) is documented elsewhere -- check which one matches your logs.
Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +99
HTTP 403 the server (HTTP 403)
magpie-crawler
A legacy crawler name, distinct from the newer Brandwatch user-agent in SEO Tools.
Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +99
HTTP 403 the server (HTTP 403)
netEstate Imprint Crawler
Not to be confused with the separately documented netEstate NE Crawler (+http://www.website-datenbank.de/), an older bot.
Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +99
Not probed —
news-please
A library default template, not a fixed signature.
Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +99
HTTP 403 the server (HTTP 403)
NewsNow Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +99
HTTP 403 the server (HTTP 403)
NovaAct Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +99
HTTP 403 the server (HTTP 403)
omgili Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +99
HTTP 403 the server (HTTP 403)
omgilibot
An older omgilibot/0.3 +http://www.omgili.com/Crawler.html form is also documented.
Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +99
HTTP 403 the server (HTTP 403)
Poseidon Research Crawler
Only the bare identifier is documented.
Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +99
Not probed —
QualifiedBot Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +99
HTTP 403 the server (HTTP 403)
Scrapy
The library default template; the version tracks the installed Scrapy release.
Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +99
HTTP 403 the server (HTTP 403)
SeekportBot Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +99
HTTP 403 the server (HTTP 403)
Seekr
alias — same bot as another row
No published user-agent string.
Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +99
Not probed —
SeekrBot
No published user-agent string.
Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +99
Not probed —
TaraGroup Intelligent Bot Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +99
HTTP 403 the server (HTTP 403)
Timpibot Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +99
HTTP 403 the server (HTTP 403)
Turnitin Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +99
HTTP 403 the server (HTTP 403)
VelenPublicWebCrawler Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +99
HTTP 403 the server (HTTP 403)
Webzio-Extended Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +99
HTTP 403 the server (HTTP 403)
yacybot
The bracketed system info varies per peer; yacybot is the stable token.
Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +99
HTTP 403 the server (HTTP 403)

Section E · Measured group

Cloud Services

0 of 4 bots read your site.

Measured Nov. 16, 2026, 2:03 a.m.

Your robots.txt says yes to all 4. Your server says no to 4.

4 bots knock Cloud Services

Gate 1 Your robots.txt

All of them ride on your catch-all group (User-agent: *) — no rule in your file names one of them individually. 4 of them are allowed with exceptions — the closed paths are housekeeping only, no known content page is among them.

Gate 2 Your server, live

4 Turned away by the server AliyunSecBot HTTP 403Amazonbot HTTP 403Amzn-User HTTP 403Google-CloudVertexBot HTTP 403

What the measurement means, point by point

Worth knowing Server-robots-instructions — what robots.txt permits

CheckedThe first thing every well-behaved bot reads. A Disallow rule on the homepage stops it before it ever requests a page, so nothing further down can compensate. A group that closes some paths but not the homepage is 'allowed with exceptions' — it counts as allowed, and the exceptions are tested against your known content pages below. A bot no rule names is allowed — silence is permission — and shown as 'not indicated'.

Foundall 4 are allowed by robots.txt; 4 of them are allowed with exceptions — the closed paths are housekeeping only, no known content page is among them.

Worth knowing Server-crawl-live-test — the live HEAD request

CheckedWhat your server really answers to one HEAD request wearing that bot's name. A firewall, a CDN rule or a bot-protection product can refuse a crawler your robots.txt welcomes — and robots.txt alone can never reveal it.

Found4 forbidden: AliyunSecBot, Amazonbot, Amzn-User, Google-CloudVertexBot

Worth knowingRead this one before acting: an ordinary desktop browser was refused too, so your server is not singling out crawlers — it is refusing our test requests wholesale (a rate limit, a geo or IP rule, or a challenge page). Check what your firewall does with unfamiliar visitors before changing any bot rule.

Worth knowing Do the two tests agree?

CheckedWhether the server-robots-instructions and the server-crawl-live-test tell the same story. They often do not: robots.txt is a published promise, the live answer is what actually happens, and only the comparison shows which one to believe.

Found4 bot(s) invited by robots.txt but refused by the server — the same ones the live test above names

Worth knowingA contradiction is worth resolving whichever way you meant it. Invited-but-refused: your firewall overrules your robots.txt — align them, or accept that the file is advertising a welcome the server does not honour. Blocked-but-served: your robots.txt line is a polite request only; a bot that ignores it gets the page, so add a server rule if the block really matters.

The Cloud Services group, bot by bot

  1. ✗

    AliyunSecBot

    robots.txt: Allowed with exceptions · your server: HTTP 403

  2. ✗

    Amazonbot

    robots.txt: Allowed with exceptions · your server: HTTP 403

  3. ✗

    Amzn-User

    robots.txt: Allowed with exceptions · your server: HTTP 403

  4. ✗

    Google-CloudVertexBot

    robots.txt: Allowed with exceptions · your server: HTTP 403

See the roster table
Every bot of the Cloud Services group, with its two verdicts
Bot Gate 1 · your robots.txt Gate 2 · your server, live What stopped it
AliyunSecBot
An unusual format, but the canonical documented instance.
Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +99
HTTP 403 the server (HTTP 403)
Amazonbot
Real instances wrap this inside a full desktop or mobile Chrome-style user-agent.
Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +99
HTTP 403 the server (HTTP 403)
Amzn-User Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +99
HTTP 403 the server (HTTP 403)
Google-CloudVertexBot Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +99
HTTP 403 the server (HTTP 403)

Section F · Measured group

Other Agents

3 of 5 bots read your site.

Measured Nov. 16, 2026, 2:03 a.m.

Your robots.txt says yes to all 5. Your server says no to 1.

5 bots knock Other Agents

Gate 1 Your robots.txt

All of them ride on your catch-all group (User-agent: *) — no rule in your file names one of them individually. 5 of them are allowed with exceptions — the closed paths are housekeeping only, no known content page is among them.

Gate 2 Your server, live

ServedRefusedNot probed

3 Read your site archive.org_botia_archiveria_archiver-web.archive.org
1 Turned away by the server TurnitinBot HTTP 403
1 Judged on the file alone ViennaTinyBot

Their operator publishes no user-agent string, so no honest knock is possible.

What the measurement means, point by point

Worth knowing Server-robots-instructions — what robots.txt permits

CheckedThe first thing every well-behaved bot reads. A Disallow rule on the homepage stops it before it ever requests a page, so nothing further down can compensate. A group that closes some paths but not the homepage is 'allowed with exceptions' — it counts as allowed, and the exceptions are tested against your known content pages below. A bot no rule names is allowed — silence is permission — and shown as 'not indicated'.

Foundall 5 are allowed by robots.txt; 5 of them are allowed with exceptions — the closed paths are housekeeping only, no known content page is among them.

Worth knowing Server-crawl-live-test — the live HEAD request

CheckedWhat your server really answers to one HEAD request wearing that bot's name. A firewall, a CDN rule or a bot-protection product can refuse a crawler your robots.txt welcomes — and robots.txt alone can never reveal it.

Found1 forbidden: TurnitinBot

Worth knowingRead this one before acting: an ordinary desktop browser was refused too, so your server is not singling out crawlers — it is refusing our test requests wholesale (a rate limit, a geo or IP rule, or a challenge page). Check what your firewall does with unfamiliar visitors before changing any bot rule.

Worth knowing Indexing directives in the headers

CheckedThe X-Robots-Tag response header of the answers your server actually served. A noindex here lets the bot in and then tells it to throw the page away — the most expensive way to be invisible, because it looks fine in every other check. Only served answers are read: a wall's or an error page's header is not yours.

Foundno X-Robots-Tag header in any served answer

Worth knowing Do the two tests agree?

CheckedWhether the server-robots-instructions and the server-crawl-live-test tell the same story. They often do not: robots.txt is a published promise, the live answer is what actually happens, and only the comparison shows which one to believe.

Found1 bot(s) invited by robots.txt but refused by the server — the same ones the live test above names

Worth knowingA contradiction is worth resolving whichever way you meant it. Invited-but-refused: your firewall overrules your robots.txt — align them, or accept that the file is advertising a welcome the server does not honour. Blocked-but-served: your robots.txt line is a polite request only; a bot that ignores it gets the page, so add a server rule if the block really matters.

Worth knowing Judged on robots.txt alone

CheckedSome operators publish a robots.txt token but never a literal user-agent string. Those bots cannot be impersonated honestly, so they are judged on your rules only — no live request was made in their name.

Found1 of 5: ViennaTinyBot

Worth knowingNothing to do. Their robots.txt verdict is still exact — it is read from your own file — only the server-side half is unknown.

The Other Agents group, bot by bot

  1. ✓

    archive.org_bot

    robots.txt: Allowed with exceptions · your server: Redirected

  2. ✓

    ia_archiver

    robots.txt: Allowed with exceptions · your server: Redirected

  3. ✓

    ia_archiver-web.archive.org

    robots.txt: Allowed with exceptions · your server: Redirected

  4. ✗

    TurnitinBot

    robots.txt: Allowed with exceptions · your server: HTTP 403

  5. !

    ViennaTinyBot

    robots.txt: Allowed with exceptions · your server: Not probed

See the roster table
Every bot of the Other Agents group, with its two verdicts
Bot Gate 1 · your robots.txt Gate 2 · your server, live What stopped it
archive.org_bot
Heritrix-version-qualified variants are also documented.
Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +99
Redirected —
ia_archiver
The historical Alexa string. Alexa retired in 2022; the string is kept for legacy matching.
Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +99
Redirected —
ia_archiver-web.archive.org
A distinct token used by the Wayback Machine's live-web / Save Page Now fetcher.
Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +99
Redirected —
TurnitinBot
A newer TurnitinBot/2.x is reported but not independently confirmed.
Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +99
HTTP 403 the server (HTTP 403)
ViennaTinyBot
Only a bare lowercase identifier appears in aggregator directories.
Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +99
Not probed —

Add to your ToDo list

Send feedback