Start free

You are exploring AkOOre's demo website — read-only, nothing you change is saved.

Crawler access — originaluniqueart.com — generated on Oct. 17, 2026, 12:36 a.m. by AkOOre

Overview · Crawler access · originaluniqueart.com · 12 Oct 2026

Your server turns away 99 of the 157 named bots that knock.

Your robots.txt lets 154 of 157 bots in, and your server served 27 of the 126 it could be asked. The access score is 57/100, with 4 critical checks and 3 to improve. 100 bots get opposite answers from your robots.txt and your server. A plain browser was refused too, so these refusals are not about crawlers.

Choose a section Or start here: A. Work orders
Allowed in robots.txt
154/ 157
Requests allowed
27/ 126
Search engines welcome
10/ 27
AI answer engines welcome
23/ 33
Requests forbidden
99

In this report

15 sections
157 bots · 8 groups · 39 checks
originaluniqueart.com · 12 Oct 2026

AStart here · Work orders

4 critical fixes, 25 in all.

Every change this report asks for — a robots.txt line or a server rule each — worst first.

  1. Server-crawl-live-test — the live HEAD request
  2. The bots that decide the outcome
  3. Server-crawl-live-test — the live HEAD request
  4. The bots that decide the outcome
25work orders(bad news) Read the work orders BWho can actually read your site

99 of 157 bots turned away.

Every named bot through your two doors: what robots.txt says about it, then what your server answered.

99turned away by the server(bad news) Open B CThe figures and the access standing

57/100, 4 critical checks.

The access score on its scale, and the counts of bots allowed, welcomed and forbidden.

Criticalproblems to fix Open C DEvery group, and which door it is behind

Lowest scored group: Google Bots, 30%.

One row per bot group, one column per outcome: the darker the cell, the bigger its share of the group.

30%lowest scored group(bad news) Open D EThe posture we measured

100 contradictions between file and server.

The door policy your site actually runs, what it buys and costs you, and where your two rulebooks disagree.

100contradictions(bad news) Open E
FSearch Engines

3 of 27 bots read your site.

The group's two doors, each point checked, then every bot one by one with what stopped it.

Criticalproblems to fix Open F GAI Bots

14 of 33 bots read your site.

The group's two doors, each point checked, then every bot one by one with what stopped it.

Goodproblems to fix Open G HGoogle Bots

0 of 10 bots read your site.

The group's two doors, each point checked, then every bot one by one with what stopped it.

Criticalproblems to fix Open H ISocial Bots

7 of 11 bots read your site.

The group's two doors, each point checked, then every bot one by one with what stopped it.

Improvableproblems to fix Open I JSEO Tools

0 of 21 bots read your site.

The group's two doors, each point checked, then every bot one by one with what stopped it.

0 / 21bots served Open J KScrapers

1 of 46 bots read your site.

The group's two doors, each point checked, then every bot one by one with what stopped it.

1 / 46bots served Open K LCloud Services

0 of 4 bots read your site.

The group's two doors, each point checked, then every bot one by one with what stopped it.

0 / 4bots served Open L MOther Agents

2 of 5 bots read your site.

The group's two doors, each point checked, then every bot one by one with what stopped it.

2 / 5bots served Open M NConclusion

Read from 157 measured bots.

Which doors are open, which are shut, which closures cost you something, and what to change first.

157bots measured Open N OWords and questions

7 terms, 10 questions answered.

The words this report uses, and the questions owners ask about robots.txt, firewalls and AI bots.

10questions answered Open O

Section A / OWork orders

Work orders

←
Guide

This section answers

What do I change, and in what order?

Evidence

Fix first — critical 4

  1. Server-crawl-live-test — the live HEAD request (H. Google Bots — this check scores 0/100)
    What to do: Read this one before acting: an ordinary desktop browser was refused too, so your server is not singling out crawlers — it is refusing our test requests wholesale (a rate limit, a geo or IP rule, or a challenge page). Check what your firewall does with unfamiliar visitors before changing any bot rule.
    What we measured on your site: 8 forbidden: Gemini-Deep-Research, Google-InspectionTool, GoogleOther, GoogleOther-Image, GoogleOther-Video, Googlebot-Image, Googlebot-Video, Storebot-Google
    Why it matters: What your server really answers to one HEAD request wearing that bot's name. A firewall, a CDN rule or a bot-protection product can refuse a crawler your robots.txt welcomes — and robots.txt alone can never reveal it.
    Where to see the detail: the H. Google Bots section shows this check with the data it is based on.
  2. The bots that decide the outcome (H. Google Bots — this check scores 0/100)
    What to do: Fix these before anything else in this report: each one is a whole audience that cannot see the site today. The table below names the exact rule or answer responsible for each.
    What we measured on your site: Google-InspectionTool is stopped by the server (HTTP 403); GoogleOther is stopped by a bot wall instead of the page — redirected to a captcha page; Googlebot-Image is stopped by its own robots.txt ru…
    Why it matters: Not every bot in this group is worth the same. These are the ones whose verdict changes what your business gets: the indexes people actually search and the assistants they actually ask.
    Where to see the detail: the H. Google Bots section shows this check with the data it is based on.
  3. Server-crawl-live-test — the live HEAD request (F. Search Engines — this check scores 15/100)
    What to do: Read this one before acting: an ordinary desktop browser was refused too, so your server is not singling out crawlers — it is refusing our test requests wholesale (a rate limit, a geo or IP rule, or a challenge page). Check what your firewall does with unfamiliar visitors before changing any bot rule.
    What we measured on your site: 17 forbidden: Amzn-SearchBot, Applebot, AspiegelBot, Baiduspider, BingPreview, Googlebot, MicrosoftPreview, MojeekBot and 9 more
    Why it matters: What your server really answers to one HEAD request wearing that bot's name. A firewall, a CDN rule or a bot-protection product can refuse a crawler your robots.txt welcomes — and robots.txt alone can never reveal it.
    Where to see the detail: the F. Search Engines section shows this check with the data it is based on.
  4. The bots that decide the outcome (F. Search Engines — this check scores 33/100)
    What to do: Fix these before anything else in this report: each one is a whole audience that cannot see the site today. The table below names the exact rule or answer responsible for each.
    What we measured on your site: Applebot is stopped by the server (HTTP 403); Baiduspider is stopped by the server (HTTP 403); Googlebot is stopped by a bot wall instead of the page — redirected to a captcha page; YandexBot is stop…
    Why it matters: Not every bot in this group is worth the same. These are the ones whose verdict changes what your business gets: the indexes people actually search and the assistants they actually ask.
    Where to see the detail: the F. Search Engines section shows this check with the data it is based on.

Then improve 3

  1. Server-crawl-live-test — the live HEAD request (G. AI Bots — this check scores 58/100)
    What to do: Read this one before acting: an ordinary desktop browser was refused too, so your server is not singling out crawlers — it is refusing our test requests wholesale (a rate limit, a geo or IP rule, or a challenge page). Check what your firewall does with unfamiliar visitors before changing any bot rule.
    What we measured on your site: 10 forbidden: AI2Bot, Ai2Bot-Dolma, DeepSeekBot, DigitalOceanGenAICrawler, DuckAssistBot, LINER Bot, PanguBot, SBIntuitionsBot and 2 more
    Why it matters: What your server really answers to one HEAD request wearing that bot's name. A firewall, a CDN rule or a bot-protection product can refuse a crawler your robots.txt welcomes — and robots.txt alone can never reveal it.
    Where to see the detail: the G. AI Bots section shows this check with the data it is based on.
  2. The bots that decide the outcome (I. Social Bots — this check scores 60/100)
    What to do: Fix these before anything else in this report: each one is a whole audience that cannot see the site today. The table below names the exact rule or answer responsible for each.
    What we measured on your site: Twitterbot is stopped by its own robots.txt rule (User-agent: twitterbot); facebookexternalhit is stopped by its own robots.txt rule (User-agent: facebookexternalhit)
    Why it matters: Not every bot in this group is worth the same. These are the ones whose verdict changes what your business gets: the indexes people actually search and the assistants they actually ask.
    Where to see the detail: the I. Social Bots section shows this check with the data it is based on.
  3. Server-crawl-live-test — the live HEAD request (I. Social Bots — this check scores 64/100)
    What to do: Read this one before acting: an ordinary desktop browser was refused too, so your server is not singling out crawlers — it is refusing our test requests wholesale (a rate limit, a geo or IP rule, or a challenge page). Check what your firewall does with unfamiliar visitors before changing any bot rule.
    What we measured on your site: 4 forbidden: Meta-ExternalAds, Meta-ExternalFetcher, Meta-WebIndexer, Quora-Bot
    Why it matters: What your server really answers to one HEAD request wearing that bot's name. A firewall, a CDN rule or a bot-protection product can refuse a crawler your robots.txt welcomes — and robots.txt alone can never reveal it.
    Where to see the detail: the I. Social Bots section shows this check with the data it is based on.

Worth knowing 18

  1. Do the two tests agree? (F. Search Engines)
    What to do with it: A contradiction is worth resolving whichever way you meant it. Invited-but-refused: your firewall overrules your robots.txt — align them, or accept that the file is advertising a welcome the server does not honour. Blocked-but-served: your robots.txt line is a polite request only; a bot that ignores it gets the page, so add a server rule if the block really matters.
    What the data says: 17 bot(s) invited by robots.txt but refused by the server — the same ones the live test above names
    Why it matters: Whether the server-robots-instructions and the server-crawl-live-test tell the same story. They often do not: robots.txt is a published promise, the live answer is what actually happens, and only the comparison shows which one to believe.
    Where to see the detail: the F. Search Engines section shows this check with the data it is based on.
  2. Judged on robots.txt alone (F. Search Engines)
    What to do with it: Nothing to do. Their robots.txt verdict is still exact — it is read from your own file — only the server-side half is unknown.
    What the data says: 7 of 27: Applebot-Extended, BingVideoPreview, Yahoo-Blogs, YahooSeeker, Yandex, YandexAdditionalBot, baidu
    Why it matters: Some operators publish a robots.txt token but never a literal user-agent string. Those bots cannot be impersonated honestly, so they are judged on your rules only — no live request was made in their name.
    Where to see the detail: the F. Search Engines section shows this check with the data it is based on.
  3. Do the two tests agree? (G. AI Bots)
    What to do with it: A contradiction is worth resolving whichever way you meant it. Invited-but-refused: your firewall overrules your robots.txt — align them, or accept that the file is advertising a welcome the server does not honour. Blocked-but-served: your robots.txt line is a polite request only; a bot that ignores it gets the page, so add a server rule if the block really matters.
    What the data says: 10 bot(s) invited by robots.txt but refused by the server — the same ones the live test above names
    Why it matters: Whether the server-robots-instructions and the server-crawl-live-test tell the same story. They often do not: robots.txt is a published promise, the live answer is what actually happens, and only the comparison shows which one to believe.
    Where to see the detail: the G. AI Bots section shows this check with the data it is based on.
  4. Judged on robots.txt alone (G. AI Bots)
    What to do with it: Nothing to do. Their robots.txt verdict is still exact — it is read from your own file — only the server-side half is unknown.
    What the data says: 9 of 33: ChatGPT Operator, ChatGPT-User/2.0, Google-Extended, Grok, GrokBot, MistralAI-User, MyCentralAIScraperBot, Perplexity-User and 1 more
    Why it matters: Some operators publish a robots.txt token but never a literal user-agent string. Those bots cannot be impersonated honestly, so they are judged on your rules only — no live request was made in their name.
    Where to see the detail: the G. AI Bots section shows this check with the data it is based on.
  5. Do the two tests agree? (H. Google Bots)
    What to do with it: A contradiction is worth resolving whichever way you meant it. Invited-but-refused: your firewall overrules your robots.txt — align them, or accept that the file is advertising a welcome the server does not honour. Blocked-but-served: your robots.txt line is a polite request only; a bot that ignores it gets the page, so add a server rule if the block really matters.
    What the data says: 7 bot(s) invited by robots.txt but refused by the server: Gemini-Deep-Research, Google-InspectionTool, GoogleOther, GoogleOther-Image, GoogleOther-Video, Googlebot-Video, Storebot-Google
    Why it matters: Whether the server-robots-instructions and the server-crawl-live-test tell the same story. They often do not: robots.txt is a published promise, the live answer is what actually happens, and only the comparison shows which one to believe.
    Where to see the detail: the H. Google Bots section shows this check with the data it is based on.
  6. Judged on robots.txt alone (H. Google Bots)
    What to do with it: Nothing to do. Their robots.txt verdict is still exact — it is read from your own file — only the server-side half is unknown.
    What the data says: 2 of 10: Googlebot-Discovery, Googlebot-News
    Why it matters: Some operators publish a robots.txt token but never a literal user-agent string. Those bots cannot be impersonated honestly, so they are judged on your rules only — no live request was made in their name.
    Where to see the detail: the H. Google Bots section shows this check with the data it is based on.
  7. Do the two tests agree? (I. Social Bots)
    What to do with it: A contradiction is worth resolving whichever way you meant it. Invited-but-refused: your firewall overrules your robots.txt — align them, or accept that the file is advertising a welcome the server does not honour. Blocked-but-served: your robots.txt line is a polite request only; a bot that ignores it gets the page, so add a server rule if the block really matters.
    What the data says: 4 bot(s) invited by robots.txt but refused by the server — the same ones the live test above names; 2 bot(s) blocked in robots.txt but served anyway: Twitterbot, facebookexternalhit
    Why it matters: Whether the server-robots-instructions and the server-crawl-live-test tell the same story. They often do not: robots.txt is a published promise, the live answer is what actually happens, and only the comparison shows which one to believe.
    Where to see the detail: the I. Social Bots section shows this check with the data it is based on.
  8. Server-crawl-live-test — the live HEAD request (J. SEO Tools)
    What to do with it: Read this one before acting: an ordinary desktop browser was refused too, so your server is not singling out crawlers — it is refusing our test requests wholesale (a rate limit, a geo or IP rule, or a challenge page). Check what your firewall does with unfamiliar visitors before changing any bot rule.
    What the data says: 20 forbidden: AhrefsBot, AudigentAdBot, AwarioRssBot, AwarioSmartBot, Brandwatch, Chrome-Lighthouse, DataForSeoBot, DotBot and 12 more
    Why it matters: What your server really answers to one HEAD request wearing that bot's name. A firewall, a CDN rule or a bot-protection product can refuse a crawler your robots.txt welcomes — and robots.txt alone can never reveal it.
    Where to see the detail: the J. SEO Tools section shows this check with the data it is based on.
  9. Do the two tests agree? (J. SEO Tools)
    What to do with it: A contradiction is worth resolving whichever way you meant it. Invited-but-refused: your firewall overrules your robots.txt — align them, or accept that the file is advertising a welcome the server does not honour. Blocked-but-served: your robots.txt line is a polite request only; a bot that ignores it gets the page, so add a server rule if the block really matters.
    What the data says: 20 bot(s) invited by robots.txt but refused by the server — the same ones the live test above names
    Why it matters: Whether the server-robots-instructions and the server-crawl-live-test tell the same story. They often do not: robots.txt is a published promise, the live answer is what actually happens, and only the comparison shows which one to believe.
    Where to see the detail: the J. SEO Tools section shows this check with the data it is based on.
  10. Judged on robots.txt alone (J. SEO Tools)
    What to do with it: Nothing to do. Their robots.txt verdict is still exact — it is read from your own file — only the server-side half is unknown.
    What the data says: 1 of 21: NeticleBot
    Why it matters: Some operators publish a robots.txt token but never a literal user-agent string. Those bots cannot be impersonated honestly, so they are judged on your rules only — no live request was made in their name.
    Where to see the detail: the J. SEO Tools section shows this check with the data it is based on.
  11. Server-crawl-live-test — the live HEAD request (K. Scrapers)
    What to do with it: Read this one before acting: an ordinary desktop browser was refused too, so your server is not singling out crawlers — it is refusing our test requests wholesale (a rate limit, a geo or IP rule, or a challenge page). Check what your firewall does with unfamiliar visitors before changing any bot rule.
    What the data says: 34 forbidden: Arquivo-web-crawler, BLEXBot, Barkrowler, Bytespider, Cotoyogi, Crawl4AI, Diffbot, EchoboxBot and 26 more
    Why it matters: What your server really answers to one HEAD request wearing that bot's name. A firewall, a CDN rule or a bot-protection product can refuse a crawler your robots.txt welcomes — and robots.txt alone can never reveal it.
    Where to see the detail: the K. Scrapers section shows this check with the data it is based on.
  12. Do the two tests agree? (K. Scrapers)
    What to do with it: A contradiction is worth resolving whichever way you meant it. Invited-but-refused: your firewall overrules your robots.txt — align them, or accept that the file is advertising a welcome the server does not honour. Blocked-but-served: your robots.txt line is a polite request only; a bot that ignores it gets the page, so add a server rule if the block really matters.
    What the data says: 34 bot(s) invited by robots.txt but refused by the server — the same ones the live test above names
    Why it matters: Whether the server-robots-instructions and the server-crawl-live-test tell the same story. They often do not: robots.txt is a published promise, the live answer is what actually happens, and only the comparison shows which one to believe.
    Where to see the detail: the K. Scrapers section shows this check with the data it is based on.
  13. Judged on robots.txt alone (K. Scrapers)
    What to do with it: Nothing to do. Their robots.txt verdict is still exact — it is read from your own file — only the server-side half is unknown.
    What the data says: 11 of 46: Amazon Kendra, Bravest, Crawlspace, Echobot Bot, JenkersBot, Poseidon Research Crawler, Seekr, SeekrBot and 3 more
    Why it matters: Some operators publish a robots.txt token but never a literal user-agent string. Those bots cannot be impersonated honestly, so they are judged on your rules only — no live request was made in their name.
    Where to see the detail: the K. Scrapers section shows this check with the data it is based on.
  14. Server-crawl-live-test — the live HEAD request (L. Cloud Services)
    What to do with it: Read this one before acting: an ordinary desktop browser was refused too, so your server is not singling out crawlers — it is refusing our test requests wholesale (a rate limit, a geo or IP rule, or a challenge page). Check what your firewall does with unfamiliar visitors before changing any bot rule.
    What the data says: 4 forbidden: AliyunSecBot, Amazonbot, Amzn-User, Google-CloudVertexBot
    Why it matters: What your server really answers to one HEAD request wearing that bot's name. A firewall, a CDN rule or a bot-protection product can refuse a crawler your robots.txt welcomes — and robots.txt alone can never reveal it.
    Where to see the detail: the L. Cloud Services section shows this check with the data it is based on.
  15. Do the two tests agree? (L. Cloud Services)
    What to do with it: A contradiction is worth resolving whichever way you meant it. Invited-but-refused: your firewall overrules your robots.txt — align them, or accept that the file is advertising a welcome the server does not honour. Blocked-but-served: your robots.txt line is a polite request only; a bot that ignores it gets the page, so add a server rule if the block really matters.
    What the data says: 4 bot(s) invited by robots.txt but refused by the server — the same ones the live test above names
    Why it matters: Whether the server-robots-instructions and the server-crawl-live-test tell the same story. They often do not: robots.txt is a published promise, the live answer is what actually happens, and only the comparison shows which one to believe.
    Where to see the detail: the L. Cloud Services section shows this check with the data it is based on.
  16. Server-crawl-live-test — the live HEAD request (M. Other Agents)
    What to do with it: Read this one before acting: an ordinary desktop browser was refused too, so your server is not singling out crawlers — it is refusing our test requests wholesale (a rate limit, a geo or IP rule, or a challenge page). Check what your firewall does with unfamiliar visitors before changing any bot rule.
    What the data says: 2 forbidden: TurnitinBot, archive.org_bot
    Why it matters: What your server really answers to one HEAD request wearing that bot's name. A firewall, a CDN rule or a bot-protection product can refuse a crawler your robots.txt welcomes — and robots.txt alone can never reveal it.
    Where to see the detail: the M. Other Agents section shows this check with the data it is based on.
  17. Do the two tests agree? (M. Other Agents)
    What to do with it: A contradiction is worth resolving whichever way you meant it. Invited-but-refused: your firewall overrules your robots.txt — align them, or accept that the file is advertising a welcome the server does not honour. Blocked-but-served: your robots.txt line is a polite request only; a bot that ignores it gets the page, so add a server rule if the block really matters.
    What the data says: 2 bot(s) invited by robots.txt but refused by the server — the same ones the live test above names
    Why it matters: Whether the server-robots-instructions and the server-crawl-live-test tell the same story. They often do not: robots.txt is a published promise, the live answer is what actually happens, and only the comparison shows which one to believe.
    Where to see the detail: the M. Other Agents section shows this check with the data it is based on.
  18. Judged on robots.txt alone (M. Other Agents)
    What to do with it: Nothing to do. Their robots.txt verdict is still exact — it is read from your own file — only the server-side half is unknown.
    What the data says: 1 of 5: ViennaTinyBot
    Why it matters: Some operators publish a robots.txt token but never a literal user-agent string. Those bots cannot be impersonated honestly, so they are judged on your rules only — no live request was made in their name.
    Where to see the detail: the M. Other Agents section shows this check with the data it is based on.

Section B / OVerdict

Who can actually read your site

Guide

This section answers

Of the 157 named bots that knock, which ones reach a page?

Terms used

robots.txtuser-agentcontrol request

Evidence

157 bots tested 8 groups 126 probed live with their own user-agent Oct. 12, 2026, 2:04 a.m. last probe

Your robots.txt lets 154 of 157 in. Your server says no to 99.

157 bots knock originaluniqueart.com

Gate 1 Your robots.txt

Allowed with exceptionsNot allowed

Your file names and blocks Googlebot-Image, Twitterbot, facebookexternalhit. 154 of them are allowed with exceptions — the closed paths are housekeeping only, no known content page is among them.

Gate 2 Your server, live

ServedCaptcha wallRefusedNot probed

27 Read your site ChatGPT-UserClaude-CodeClaude-SearchBotClaude-UserClaude-WebClaudeBotGPTBotMistralAI-User/1.0OAI-SearchBotPerplexity-User/1.0 + 17 more
99 Turned away by the server AI2Bot HTTP 403Ai2Bot-Dolma HTTP 403DeepSeekBot HTTP 403DigitalOceanGenAICrawler HTTP 403DuckAssistBot HTTP 403LINER Bot HTTP 403PanguBot HTTP 403SBIntuitionsBot HTTP 403cohere-training-data-crawler HTTP 403img2dataset HTTP 403 + 89 more
31 Judged on the file alone ChatGPT OperatorChatGPT-User/2.0Google-ExtendedGrokGrokBotMistralAI-UserMyCentralAIScraperBotPerplexity-Userquillbot.comGooglebot-Discovery + 21 more

Their operator publishes no user-agent string, so no honest knock is possible.

Instrument check

HTTP 403 — a plain browser was refused too

Sent as Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/140.0.0.0 Safari/537.36

Section C / OScores

The figures and the access standing

Guide

This section answers

How much of the traffic that matters gets through?

Terms used

scored checkwelcome

EvidenceCritical

57/100 Critical

05080100
Critical
4
Needs work
3
Good
5
154/157 Allowed in robots.txt
27/126 Requests allowed
10/27 Search engines welcome
23/33 AI answer engines welcome
99 Requests forbidden

Section D / OGroups

Every group, and which door it is behind

Guide

This section answers

Which group should I open next?

Evidence

Every bot group, with how many bots got each outcome
GroupScore Blocked by robots.txtServedCaptcha wallRefusedNot probed
Scored — being shut out here costs you
A. Search Engines 49% Fix soon · 3 1 16 7
B. AI Bots 81% Good · 14 · 10 9
C. Google Bots 30% Fix soon 1 · 5 3 2
D. Social Bots 69% Improvable 2 7 · 4 ·
Measured, never marked wrong
E. SEO Tools not scored · · · 20 1
F. Scrapers not scored · 1 · 34 11
G. Cloud Services not scored · · · 4 ·
H. Other Agents not scored · 2 · 2 1

Section E / ODiagnosis

The posture we measured

Guide

This section answers

Is this door policy a decision, or an accident?

Terms used

postureinvited, then refusedblocked, then served

Evidence

Strategy

Hard wall

robots.txt welcomes the bots, but the server turns most requests away (a firewall, CDN rule or bot challenge). The published rulebook and the door policy disagree — the door wins.

What it buys you

  • Strong, enforced protection: bots that ignore robots.txt are stopped anyway.
  • Legitimate if crawler verification by IP or reverse DNS lets the real search engines through a door that name-claimers cannot use.

What it costs you

  • If the wall does NOT allowlist real crawlers by IP, search engines and AI assistants are locked out with everyone else — invisibility, not protection.
  • The welcoming robots.txt is misleading — it advertises an access the server does not grant.

Where your two rulebooks contradict each other

98 invited, then refused

robots.txt welcomes AI2Bot, Ai2Bot-Dolma, DeepSeekBot, DigitalOceanGenAICrawler, DuckAssistBot, LINER Bot and 92 more (among others), but the server turns them away. The server wins — your file is advertising a welcome it does not honour.

2 blocked, then served

robots.txt forbids Twitterbot, facebookexternalhit, yet the server serves them anyway: the block only holds against bots polite enough to honour it.

Section F / OScored group

Search Engines

Guide

This section answers

Which bots of this group can read your site, and what stops the others?

Terms used

Gate 1Gate 2captcha wall

EvidenceCritical

Measured Oct. 12, 2026, 2:04 a.m.

1/3 scored points pass. Weakest point: Server-crawl-live-test — the live HEAD request (15%).

Your robots.txt says yes to all 27. Your server says no to 17.

27 bots knock A. Search Engines

Gate 1 Your robots.txt

2 of them have a group of their own in your file, 25 ride on the catch-all group (User-agent: *) and 0 are named by no rule. 27 of them are allowed with exceptions — the closed paths are housekeeping only, no known content page is among them.

Gate 2 Your server, live

ServedCaptcha wallRefusedNot probed

3 Read your site BingbotDuckDuckBotadidxbot
17 Turned away by the server Amzn-SearchBot HTTP 403Applebot HTTP 403AspiegelBot HTTP 403Baiduspider HTTP 403BingPreview HTTP 403MicrosoftPreview HTTP 403MojeekBot HTTP 403PetalBot HTTP 403SeznamBot HTTP 403SeznamHomepageCrawler HTTP 403 + 7 more
7 Judged on the file alone Applebot-ExtendedBingVideoPreviewYahoo-BlogsYahooSeekerYandexYandexAdditionalBotbaidu

Their operator publishes no user-agent string, so no honest knock is possible.

What the measurement means, point by point

Good Server-robots-instructions — what robots.txt permits 100%

CheckedThe first thing every well-behaved bot reads. A Disallow rule on the homepage stops it before it ever requests a page, so nothing further down can compensate. A group that closes some paths but not the homepage is 'allowed with exceptions' — it counts as allowed, and the exceptions are tested against your known content pages below. A bot no rule names is allowed — silence is permission — and shown as 'not indicated'.

Foundall 27 are allowed by robots.txt; 27 of them are allowed with exceptions — the closed paths are housekeeping only, no known content page is among them.

Critical Server-crawl-live-test — the live HEAD request 15%

CheckedWhat your server really answers to one HEAD request wearing that bot's name. A firewall, a CDN rule or a bot-protection product can refuse a crawler your robots.txt welcomes — and robots.txt alone can never reveal it.

Found17 forbidden: Amzn-SearchBot, Applebot, AspiegelBot, Baiduspider, BingPreview, Googlebot, MicrosoftPreview, MojeekBot and 9 more

Do thisRead this one before acting: an ordinary desktop browser was refused too, so your server is not singling out crawlers — it is refusing our test requests wholesale (a rate limit, a geo or IP rule, or a challenge page). Check what your firewall does with unfamiliar visitors before changing any bot rule.

Info Indexing directives in the headers

CheckedThe X-Robots-Tag response header of the answers your server actually served. A noindex here lets the bot in and then tells it to throw the page away — the most expensive way to be invisible, because it looks fine in every other check. Only served answers are read: a wall's or an error page's header is not yours.

Foundno X-Robots-Tag header in any served answer

Info Do the two tests agree?

CheckedWhether the server-robots-instructions and the server-crawl-live-test tell the same story. They often do not: robots.txt is a published promise, the live answer is what actually happens, and only the comparison shows which one to believe.

Found17 bot(s) invited by robots.txt but refused by the server — the same ones the live test above names

Worth knowingA contradiction is worth resolving whichever way you meant it. Invited-but-refused: your firewall overrules your robots.txt — align them, or accept that the file is advertising a welcome the server does not honour. Blocked-but-served: your robots.txt line is a polite request only; a bot that ignores it gets the page, so add a server rule if the block really matters.

Critical The bots that decide the outcome 33%

CheckedNot every bot in this group is worth the same. These are the ones whose verdict changes what your business gets: the indexes people actually search and the assistants they actually ask.

FoundApplebot is stopped by the server (HTTP 403); Baiduspider is stopped by the server (HTTP 403); Googlebot is stopped by a bot wall instead of the page — redirected to a captcha page; YandexBot is stopped by the server (HTTP 403)

Do thisFix these before anything else in this report: each one is a whole audience that cannot see the site today. The table below names the exact rule or answer responsible for each.

Info Judged on robots.txt alone

CheckedSome operators publish a robots.txt token but never a literal user-agent string. Those bots cannot be impersonated honestly, so they are judged on your rules only — no live request was made in their name.

Found7 of 27: Applebot-Extended, BingVideoPreview, Yahoo-Blogs, YahooSeeker, Yandex, YandexAdditionalBot, baidu

Worth knowingNothing to do. Their robots.txt verdict is still exact — it is read from your own file — only the server-side half is unknown.

The A. Search Engines group, bot by bot

Every bot of the A. Search Engines group, with its two verdicts
Bot server-robots-instructions server-crawl-live-test What stopped it
Amzn-SearchBot
Feeds Amazon, Alexa and Rufus search. 		Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +97 		HTTP 403 the server (HTTP 403)
Applebot
The device, OS and WebKit fields vary between Apple's crawlers. 		Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +97 		HTTP 403 the server (HTTP 403)
Applebot-Extended
A robots.txt token with no distinct user-agent: it governs whether Apple Intelligence may train on content Applebot already fetched. 		Allowed with exceptions
by User-agent: applebot-extended
closed: /admin/, /*/admin/, /manage/ +96 		Not probed —
AspiegelBot
alias — same bot as another row
The legacy name, superseded by PetalBot. 		Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +97 		HTTP 403 the server (HTTP 403)
Baiduspider Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +97 		HTTP 403 the server (HTTP 403)
BingPreview Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +97 		HTTP 403 the server (HTTP 403)
BingVideoPreview
Bing confirms this bot exists but publishes no literal user-agent string, so only its robots.txt token is checked. 		Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +97 		Not probed —
Bingbot Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +97 		Redirected —
DuckDuckBot Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +97 		Redirected —
Googlebot
The mobile variant wraps a Chrome/Android string around the same (compatible; Googlebot/2.1; ...) suffix. 		Allowed with exceptions
by User-agent: googlebot
closed: /admin/, /*/admin/, /manage/ +96 		Captcha a bot wall instead of the page — redirected to a captcha page
MicrosoftPreview Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +97 		HTTP 403 the server (HTTP 403)
MojeekBot
The version number increments over time (0.2 to 0.11+ seen). 		Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +97 		HTTP 403 the server (HTTP 403)
PetalBot
Huawei Petal Search. The mobile variant prefixes Android and AppleWebKit fields. 		Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +97 		HTTP 403 the server (HTTP 403)
SeznamBot Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +97 		HTTP 403 the server (HTTP 403)
SeznamHomepageCrawler Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +97 		HTTP 403 the server (HTTP 403)
Slurp Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +97 		HTTP 403 the server (HTTP 403)
Teoma Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +97 		HTTP 403 the server (HTTP 403)
Yahoo-Blogs
Legacy, listed only by unofficial aggregators. 		Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +97 		Not probed —
Yahoo-FeedSeeker
Historical form, not verified as current. 		Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +97 		HTTP 403 the server (HTTP 403)
Yahoo-MMCrawler
Historical form, not verified as current. 		Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +97 		HTTP 403 the server (HTTP 403)
YahooSeeker
Legacy: Yahoo Search is Bing-powered now and no current documentation publishes a string. 		Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +97 		Not probed —
Yandex
alias — same bot as another row
A robots.txt control token matching any Yandex* bot, not a distinct request user-agent. 		Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +97 		Not probed —
YandexAdditional
Mostly a robots.txt control token; this string comes from secondary sources, not from Yandex's own documentation. 		Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +97 		HTTP 403 the server (HTTP 403)
YandexAdditionalBot
alias — same bot as another row
Same bot and same token as YandexAdditional. 		Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +97 		Not probed —
YandexBot Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +97 		HTTP 403 the server (HTTP 403)
adidxbot
Bing's ad-landing-page quality crawler. 		Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +97 		Redirected —
baidu
alias — same bot as another row
The same bot as Baiduspider, not a separate name or string. 		Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +97 		Not probed —

Section G / OScored group

AI Bots

Guide

This section answers

Which bots of this group can read your site, and what stops the others?

Terms used

Gate 1Gate 2captcha wall

EvidenceGood

Measured Oct. 12, 2026, 2:04 a.m.

2/3 scored points pass. Weakest point: Server-crawl-live-test — the live HEAD request (58%).

Your robots.txt says yes to all 33. Your server says no to 10.

33 bots knock B. AI Bots

Gate 1 Your robots.txt

18 of them have a group of their own in your file, 15 ride on the catch-all group (User-agent: *) and 0 are named by no rule. 33 of them are allowed with exceptions — the closed paths are housekeeping only, no known content page is among them.

Gate 2 Your server, live

ServedRefusedNot probed

14 Read your site ChatGPT-UserClaude-CodeClaude-SearchBotClaude-UserClaude-WebClaudeBotGPTBotMistralAI-User/1.0OAI-SearchBotPerplexity-User/1.0 + 4 more
10 Turned away by the server AI2Bot HTTP 403Ai2Bot-Dolma HTTP 403DeepSeekBot HTTP 403DigitalOceanGenAICrawler HTTP 403DuckAssistBot HTTP 403LINER Bot HTTP 403PanguBot HTTP 403SBIntuitionsBot HTTP 403cohere-training-data-crawler HTTP 403img2dataset HTTP 403
9 Judged on the file alone ChatGPT OperatorChatGPT-User/2.0Google-ExtendedGrokGrokBotMistralAI-UserMyCentralAIScraperBotPerplexity-Userquillbot.com

Their operator publishes no user-agent string, so no honest knock is possible.

What the measurement means, point by point

Good Server-robots-instructions — what robots.txt permits 100%

CheckedThe first thing every well-behaved bot reads. A Disallow rule on the homepage stops it before it ever requests a page, so nothing further down can compensate. A group that closes some paths but not the homepage is 'allowed with exceptions' — it counts as allowed, and the exceptions are tested against your known content pages below. A bot no rule names is allowed — silence is permission — and shown as 'not indicated'.

Foundall 33 are allowed by robots.txt; 33 of them are allowed with exceptions — the closed paths are housekeeping only, no known content page is among them.

Needs work Server-crawl-live-test — the live HEAD request 58%

CheckedWhat your server really answers to one HEAD request wearing that bot's name. A firewall, a CDN rule or a bot-protection product can refuse a crawler your robots.txt welcomes — and robots.txt alone can never reveal it.

Found10 forbidden: AI2Bot, Ai2Bot-Dolma, DeepSeekBot, DigitalOceanGenAICrawler, DuckAssistBot, LINER Bot, PanguBot, SBIntuitionsBot and 2 more

Do thisRead this one before acting: an ordinary desktop browser was refused too, so your server is not singling out crawlers — it is refusing our test requests wholesale (a rate limit, a geo or IP rule, or a challenge page). Check what your firewall does with unfamiliar visitors before changing any bot rule.

Info Indexing directives in the headers

CheckedThe X-Robots-Tag response header of the answers your server actually served. A noindex here lets the bot in and then tells it to throw the page away — the most expensive way to be invisible, because it looks fine in every other check. Only served answers are read: a wall's or an error page's header is not yours.

Foundno X-Robots-Tag header in any served answer

Info Do the two tests agree?

CheckedWhether the server-robots-instructions and the server-crawl-live-test tell the same story. They often do not: robots.txt is a published promise, the live answer is what actually happens, and only the comparison shows which one to believe.

Found10 bot(s) invited by robots.txt but refused by the server — the same ones the live test above names

Worth knowingA contradiction is worth resolving whichever way you meant it. Invited-but-refused: your firewall overrules your robots.txt — align them, or accept that the file is advertising a welcome the server does not honour. Blocked-but-served: your robots.txt line is a polite request only; a bot that ignores it gets the page, so add a server rule if the block really matters.

Good The bots that decide the outcome 86%

CheckedNot every bot in this group is worth the same. These are the ones whose verdict changes what your business gets: the indexes people actually search and the assistants they actually ask.

FoundDuckAssistBot is stopped by the server (HTTP 403)

Info Judged on robots.txt alone

CheckedSome operators publish a robots.txt token but never a literal user-agent string. Those bots cannot be impersonated honestly, so they are judged on your rules only — no live request was made in their name.

Found9 of 33: ChatGPT Operator, ChatGPT-User/2.0, Google-Extended, Grok, GrokBot, MistralAI-User, MyCentralAIScraperBot, Perplexity-User and 1 more

Worth knowingNothing to do. Their robots.txt verdict is still exact — it is read from your own file — only the server-side half is unknown.

The B. AI Bots group, bot by bot

Every bot of the B. AI Bots group, with its two verdicts
Bot server-robots-instructions server-crawl-live-test What stopped it
AI2Bot
A bare token. 		Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +97 		HTTP 403 the server (HTTP 403)
Ai2Bot-Dolma Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +97 		HTTP 403 the server (HTTP 403)
ChatGPT Operator
alias — same bot as another row
Folded into ChatGPT agent in 2025. It sends no distinctive user-agent -- it uses an ordinary Chrome desktop string and self-identifies with a Signature-Agent: https://chatgpt.com header, so it cannot be impersonated by user-agent alone. 		Allowed with exceptions
by User-agent: chatgpt-user
closed: /admin/, /*/admin/, /manage/ +96 		Not probed —
ChatGPT-User
OpenAI's live fetch when a ChatGPT user asks about your page. 		Allowed with exceptions
by User-agent: chatgpt-user
closed: /admin/, /*/admin/, /manage/ +96 		Redirected —
ChatGPT-User/2.0
alias — same bot as another row
OpenAI's published version is 1.0; there is no evidence of a 2.0, so only the token is checked. 		Allowed with exceptions
by User-agent: chatgpt-user
closed: /admin/, /*/admin/, /manage/ +96 		Not probed —
Claude-Code
alias — same bot as another row
Not a separately documented crawler: the Claude Code CLI's own fetches ride the Claude-User pattern. 		Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +97 		Redirected —
Claude-SearchBot Allowed with exceptions
by User-agent: claude-searchbot
closed: /admin/, /*/admin/, /manage/ +96 		Redirected —
Claude-User Allowed with exceptions
by User-agent: claude-user
closed: /admin/, /*/admin/, /manage/ +96 		Redirected —
Claude-Web
alias — same bot as another row
Deprecated, replaced by ClaudeBot / Claude-User / Claude-SearchBot. The string is historical. 		Allowed with exceptions
by User-agent: claude-web
closed: /admin/, /*/admin/, /manage/ +96 		Redirected —
ClaudeBot Allowed with exceptions
by User-agent: claudebot
closed: /admin/, /*/admin/, /manage/ +96 		Redirected —
DeepSeekBot Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +97 		HTTP 403 the server (HTTP 403)
DigitalOceanGenAICrawler
Names conflict: DigitalOcean's current docs say DigitalOceanGradientAICrawler/1.0, a widely cited third-party report says DigitalOceanGenAICrawler/1.0. 		Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +97 		HTTP 403 the server (HTTP 403)
DuckAssistBot Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +97 		HTTP 403 the server (HTTP 403)
GPTBot Allowed with exceptions
by User-agent: gptbot
closed: /admin/, /*/admin/, /manage/ +96 		Redirected —
Google-Extended
A robots.txt token only: the crawl happens under the standard Googlebot or GoogleOther user-agent. It governs whether your pages may ground Gemini answers and AI Overviews. 		Allowed with exceptions
by User-agent: google-extended
closed: /admin/, /*/admin/, /manage/ +96 		Not probed —
Grok
xAI publishes no crawler documentation page; third-party trackers disagree (Grok, xAI-Grok/1.0, GrokBot/1.0), so no string is sent. 		Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +97 		Not probed —
GrokBot
alias — same bot as another row
Same caveat as Grok: no official documentation. 		Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +97 		Not probed —
LINER Bot Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +97 		HTTP 403 the server (HTTP 403)
MistralAI-User
alias — same bot as another row
1.0 is the only published version. 		Allowed with exceptions
by User-agent: mistralai-user
closed: /admin/, /*/admin/, /manage/ +96 		Not probed —
MistralAI-User/1.0 Allowed with exceptions
by User-agent: mistralai-user
closed: /admin/, /*/admin/, /manage/ +96 		Redirected —
MyCentralAIScraperBot
The token circulates in blocklists; the operator and the user-agent string are undocumented. 		Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +97 		Not probed —
OAI-SearchBot
Builds the index behind ChatGPT search results. 		Allowed with exceptions
by User-agent: oai-searchbot
closed: /admin/, /*/admin/, /manage/ +96 		Redirected —
PanguBot
Huawei's Pangu LLM crawler. 		Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +97 		HTTP 403 the server (HTTP 403)
Perplexity-User
alias — same bot as another row
Same bot as Perplexity-User/1.0. 		Allowed with exceptions
by User-agent: perplexity-user
closed: /admin/, /*/admin/, /manage/ +96 		Not probed —
Perplexity-User/1.0 Allowed with exceptions
by User-agent: perplexity-user
closed: /admin/, /*/admin/, /manage/ +96 		Redirected —
PerplexityBot Allowed with exceptions
by User-agent: perplexitybot
closed: /admin/, /*/admin/, /manage/ +96 		Redirected —
SBIntuitionsBot Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +97 		HTTP 403 the server (HTTP 403)
YouBot Allowed with exceptions
by User-agent: youbot
closed: /admin/, /*/admin/, /manage/ +96 		Redirected —
anthropic-ai
alias — same bot as another row
A deprecated legacy token, superseded by ClaudeBot. 		Allowed with exceptions
by User-agent: anthropic-ai
closed: /admin/, /*/admin/, /manage/ +96 		Redirected —
cohere-ai
A bare token; no Mozilla-wrapped form is documented. 		Allowed with exceptions
by User-agent: cohere-ai
closed: /admin/, /*/admin/, /manage/ +96 		Redirected —
cohere-training-data-crawler Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +97 		HTTP 403 the server (HTTP 403)
img2dataset
An open-source tool run by many independent operators. 		Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +97 		HTTP 403 the server (HTTP 403)
quillbot.com
The robots.txt token is confirmed; no literal user-agent string is published. 		Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +97 		Not probed —

Section H / OScored group

Google Bots

Guide

This section answers

Which bots of this group can read your site, and what stops the others?

Terms used

Gate 1Gate 2captcha wall

EvidenceCritical

Measured Oct. 12, 2026, 2:04 a.m.

1/3 scored points pass. Weakest point: Server-crawl-live-test — the live HEAD request (0%).

Your robots.txt lets 9 of 10 in. Your server says no to 8.

10 bots knock C. Google Bots

Gate 1 Your robots.txt

Allowed with exceptionsNot allowed

Your file names and blocks Googlebot-Image. 9 of them are allowed with exceptions — the closed paths are housekeeping only, no known content page is among them.

Gate 2 Your server, live

Captcha wallRefusedNot probed

8 Turned away by the server Gemini-Deep-Research HTTP 403Google-InspectionTool HTTP 403Storebot-Google HTTP 403GoogleOther CaptchaGoogleOther-Image CaptchaGoogleOther-Video CaptchaGooglebot-Image CaptchaGooglebot-Video Captcha
2 Judged on the file alone Googlebot-DiscoveryGooglebot-News

Their operator publishes no user-agent string, so no honest knock is possible.

What the measurement means, point by point

Good Server-robots-instructions — what robots.txt permits 90%

CheckedThe first thing every well-behaved bot reads. A Disallow rule on the homepage stops it before it ever requests a page, so nothing further down can compensate. A group that closes some paths but not the homepage is 'allowed with exceptions' — it counts as allowed, and the exceptions are tested against your known content pages below. A bot no rule names is allowed — silence is permission — and shown as 'not indicated'.

Found1 of 10 not allowed: Googlebot-Image; 9 of them are allowed with exceptions — the closed paths are housekeeping only, no known content page is among them.

Critical Server-crawl-live-test — the live HEAD request 0%

CheckedWhat your server really answers to one HEAD request wearing that bot's name. A firewall, a CDN rule or a bot-protection product can refuse a crawler your robots.txt welcomes — and robots.txt alone can never reveal it.

Found8 forbidden: Gemini-Deep-Research, Google-InspectionTool, GoogleOther, GoogleOther-Image, GoogleOther-Video, Googlebot-Image, Googlebot-Video, Storebot-Google

Do thisRead this one before acting: an ordinary desktop browser was refused too, so your server is not singling out crawlers — it is refusing our test requests wholesale (a rate limit, a geo or IP rule, or a challenge page). Check what your firewall does with unfamiliar visitors before changing any bot rule.

Info Do the two tests agree?

CheckedWhether the server-robots-instructions and the server-crawl-live-test tell the same story. They often do not: robots.txt is a published promise, the live answer is what actually happens, and only the comparison shows which one to believe.

Found7 bot(s) invited by robots.txt but refused by the server: Gemini-Deep-Research, Google-InspectionTool, GoogleOther, GoogleOther-Image, GoogleOther-Video, Googlebot-Video, Storebot-Google

Worth knowingA contradiction is worth resolving whichever way you meant it. Invited-but-refused: your firewall overrules your robots.txt — align them, or accept that the file is advertising a welcome the server does not honour. Blocked-but-served: your robots.txt line is a polite request only; a bot that ignores it gets the page, so add a server rule if the block really matters.

Critical The bots that decide the outcome 0%

CheckedNot every bot in this group is worth the same. These are the ones whose verdict changes what your business gets: the indexes people actually search and the assistants they actually ask.

FoundGoogle-InspectionTool is stopped by the server (HTTP 403); GoogleOther is stopped by a bot wall instead of the page — redirected to a captcha page; Googlebot-Image is stopped by its own robots.txt rule (User-agent: googlebot-image)

Do thisFix these before anything else in this report: each one is a whole audience that cannot see the site today. The table below names the exact rule or answer responsible for each.

Info Judged on robots.txt alone

CheckedSome operators publish a robots.txt token but never a literal user-agent string. Those bots cannot be impersonated honestly, so they are judged on your rules only — no live request was made in their name.

Found2 of 10: Googlebot-Discovery, Googlebot-News

Worth knowingNothing to do. Their robots.txt verdict is still exact — it is read from your own file — only the server-side half is unknown.

The C. Google Bots group, bot by bot

Every bot of the C. Google Bots group, with its two verdicts
Bot server-robots-instructions server-crawl-live-test What stopped it
Gemini-Deep-Research Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +97 		HTTP 403 the server (HTTP 403)
Google-InspectionTool
What Search Console's URL Inspection uses. A mobile variant wraps it in a Nexus 5X Chrome string. 		Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +97 		HTTP 403 the server (HTTP 403)
GoogleOther
A bare token; may also appear via standard Googlebot-style strings. 		Allowed with exceptions
by User-agent: googleother
closed: /admin/, /*/admin/, /manage/ +96 		Captcha a bot wall instead of the page — redirected to a captcha page
GoogleOther-Image Allowed with exceptions
by User-agent: googleother
closed: /admin/, /*/admin/, /manage/ +96 		Captcha a bot wall instead of the page — redirected to a captcha page
GoogleOther-Video Allowed with exceptions
by User-agent: googleother
closed: /admin/, /*/admin/, /manage/ +96 		Captcha a bot wall instead of the page — redirected to a captcha page
Googlebot-Discovery
alias — same bot as another row
No documentation found for this exact name. Discover surfacing uses the standard Googlebot / Googlebot-Image, so this is likely a naming confusion. 		Allowed with exceptions
by User-agent: googlebot
closed: /admin/, /*/admin/, /manage/ +96 		Not probed —
Googlebot-Image Not allowed
by User-agent: googlebot-image 		Captcha its own robots.txt rule (User-agent: googlebot-image)
Googlebot-News
alias — same bot as another row
A distinct robots.txt token with no separate string: the requests use the standard Googlebot user-agent. 		Allowed with exceptions
by User-agent: googlebot
closed: /admin/, /*/admin/, /manage/ +96 		Not probed —
Googlebot-Video Allowed with exceptions
by User-agent: googlebot
closed: /admin/, /*/admin/, /manage/ +96 		Captcha a bot wall instead of the page — redirected to a captcha page
Storebot-Google
An Android/mobile variant also exists. 		Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +97 		HTTP 403 the server (HTTP 403)

Section I / OScored group

Social Bots

Guide

This section answers

Which bots of this group can read your site, and what stops the others?

Terms used

Gate 1Gate 2captcha wall

EvidenceImprovable

Measured Oct. 12, 2026, 2:04 a.m.

1/3 scored points pass. Weakest point: The bots that decide the outcome (60%).

Your robots.txt lets 9 of 11 in. Your server says no to 4.

11 bots knock D. Social Bots

Gate 1 Your robots.txt

Allowed with exceptionsNot allowed

Your file names and blocks Twitterbot, facebookexternalhit. 9 of them are allowed with exceptions — the closed paths are housekeeping only, no known content page is among them.

Gate 2 Your server, live

ServedRefused

7 Read your site FacebookBotLinkedInBotMeta-ExternalAgentPinterestbotSlackbotTwitterbotfacebookexternalhit
4 Turned away by the server Meta-ExternalAds HTTP 403Meta-ExternalFetcher HTTP 403Meta-WebIndexer HTTP 403Quora-Bot HTTP 403

What the measurement means, point by point

Good Server-robots-instructions — what robots.txt permits 82%

CheckedThe first thing every well-behaved bot reads. A Disallow rule on the homepage stops it before it ever requests a page, so nothing further down can compensate. A group that closes some paths but not the homepage is 'allowed with exceptions' — it counts as allowed, and the exceptions are tested against your known content pages below. A bot no rule names is allowed — silence is permission — and shown as 'not indicated'.

Found2 of 11 not allowed: Twitterbot, facebookexternalhit; 9 of them are allowed with exceptions — the closed paths are housekeeping only, no known content page is among them.

Needs work Server-crawl-live-test — the live HEAD request 64%

CheckedWhat your server really answers to one HEAD request wearing that bot's name. A firewall, a CDN rule or a bot-protection product can refuse a crawler your robots.txt welcomes — and robots.txt alone can never reveal it.

Found4 forbidden: Meta-ExternalAds, Meta-ExternalFetcher, Meta-WebIndexer, Quora-Bot

Do thisRead this one before acting: an ordinary desktop browser was refused too, so your server is not singling out crawlers — it is refusing our test requests wholesale (a rate limit, a geo or IP rule, or a challenge page). Check what your firewall does with unfamiliar visitors before changing any bot rule.

Info Indexing directives in the headers

CheckedThe X-Robots-Tag response header of the answers your server actually served. A noindex here lets the bot in and then tells it to throw the page away — the most expensive way to be invisible, because it looks fine in every other check. Only served answers are read: a wall's or an error page's header is not yours.

Foundno X-Robots-Tag header in any served answer

Info Do the two tests agree?

CheckedWhether the server-robots-instructions and the server-crawl-live-test tell the same story. They often do not: robots.txt is a published promise, the live answer is what actually happens, and only the comparison shows which one to believe.

Found4 bot(s) invited by robots.txt but refused by the server — the same ones the live test above names; 2 bot(s) blocked in robots.txt but served anyway: Twitterbot, facebookexternalhit

Worth knowingA contradiction is worth resolving whichever way you meant it. Invited-but-refused: your firewall overrules your robots.txt — align them, or accept that the file is advertising a welcome the server does not honour. Blocked-but-served: your robots.txt line is a polite request only; a bot that ignores it gets the page, so add a server rule if the block really matters.

Needs work The bots that decide the outcome 60%

CheckedNot every bot in this group is worth the same. These are the ones whose verdict changes what your business gets: the indexes people actually search and the assistants they actually ask.

FoundTwitterbot is stopped by its own robots.txt rule (User-agent: twitterbot); facebookexternalhit is stopped by its own robots.txt rule (User-agent: facebookexternalhit)

Do thisFix these before anything else in this report: each one is a whole audience that cannot see the site today. The table below names the exact rule or answer responsible for each.

The D. Social Bots group, bot by bot

Every bot of the D. Social Bots group, with its two verdicts
Bot server-robots-instructions server-crawl-live-test What stopped it
FacebookBot
Meta's general research/product crawler, distinct from facebookexternalhit. 		Allowed with exceptions
by User-agent: facebookbot
closed: /admin/, /*/admin/, /manage/ +96 		Redirected —
LinkedInBot
The HttpClient version field varies. 		Allowed with exceptions
by User-agent: linkedinbot
closed: /admin/, /*/admin/, /manage/ +96 		Redirected —
Meta-ExternalAds
Ad-landing-page review. 		Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +97 		HTTP 403 the server (HTTP 403)
Meta-ExternalAgent
AI training plus search indexing. 		Allowed with exceptions
by User-agent: meta-externalagent
closed: /admin/, /*/admin/, /manage/ +96 		Redirected —
Meta-ExternalFetcher
On-demand fetch for Meta AI features. 		Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +97 		HTTP 403 the server (HTTP 403)
Meta-WebIndexer
Meta's newer (2025) general web-indexing crawler. 		Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +97 		HTTP 403 the server (HTTP 403)
Pinterestbot
A lighter fetcher is also seen: Pinterest/0.2 (+https://www.pinterest.com/bot.html). 		Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +97 		Redirected —
Quora-Bot Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +97 		HTTP 403 the server (HTTP 403)
Slackbot
Other Slack integrations use Slackbot 1.0 (+https://api.slack.com/robots). 		Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +97 		Redirected —
Twitterbot Not allowed
by User-agent: twitterbot 		Redirected its own robots.txt rule (User-agent: twitterbot)
facebookexternalhit
The link-preview fetcher; also seen bare as facebookexternalhit/1.1. 		Not allowed
by User-agent: facebookexternalhit 		Redirected its own robots.txt rule (User-agent: facebookexternalhit)

Section J / OMeasured group

SEO Tools

Guide

This section answers

Which bots of this group can read your site, and what stops the others?

Terms used

Gate 1Gate 2captcha wall

Evidence

Measured Oct. 12, 2026, 2:04 a.m.

Your robots.txt says yes to all 21. Your server says no to 20.

21 bots knock E. SEO Tools

Gate 1 Your robots.txt

All of them ride on your catch-all group (User-agent: *) — no rule in your file names one of them individually. 21 of them are allowed with exceptions — the closed paths are housekeeping only, no known content page is among them.

Gate 2 Your server, live

RefusedNot probed

20 Turned away by the server AhrefsBot HTTP 403AudigentAdBot HTTP 403AwarioRssBot HTTP 403AwarioSmartBot HTTP 403Brandwatch HTTP 403Chrome-Lighthouse HTTP 403DataForSeoBot HTTP 403DotBot HTTP 403Google Page Speed Insights HTTP 403Jetslide HTTP 403 + 10 more
1 Judged on the file alone NeticleBot

Their operator publishes no user-agent string, so no honest knock is possible.

What the measurement means, point by point

Info Server-robots-instructions — what robots.txt permits

CheckedThe first thing every well-behaved bot reads. A Disallow rule on the homepage stops it before it ever requests a page, so nothing further down can compensate. A group that closes some paths but not the homepage is 'allowed with exceptions' — it counts as allowed, and the exceptions are tested against your known content pages below. A bot no rule names is allowed — silence is permission — and shown as 'not indicated'.

Foundall 21 are allowed by robots.txt; 21 of them are allowed with exceptions — the closed paths are housekeeping only, no known content page is among them.

Info Server-crawl-live-test — the live HEAD request

CheckedWhat your server really answers to one HEAD request wearing that bot's name. A firewall, a CDN rule or a bot-protection product can refuse a crawler your robots.txt welcomes — and robots.txt alone can never reveal it.

Found20 forbidden: AhrefsBot, AudigentAdBot, AwarioRssBot, AwarioSmartBot, Brandwatch, Chrome-Lighthouse, DataForSeoBot, DotBot and 12 more

Worth knowingRead this one before acting: an ordinary desktop browser was refused too, so your server is not singling out crawlers — it is refusing our test requests wholesale (a rate limit, a geo or IP rule, or a challenge page). Check what your firewall does with unfamiliar visitors before changing any bot rule.

Info Do the two tests agree?

CheckedWhether the server-robots-instructions and the server-crawl-live-test tell the same story. They often do not: robots.txt is a published promise, the live answer is what actually happens, and only the comparison shows which one to believe.

Found20 bot(s) invited by robots.txt but refused by the server — the same ones the live test above names

Worth knowingA contradiction is worth resolving whichever way you meant it. Invited-but-refused: your firewall overrules your robots.txt — align them, or accept that the file is advertising a welcome the server does not honour. Blocked-but-served: your robots.txt line is a polite request only; a bot that ignores it gets the page, so add a server rule if the block really matters.

Info Judged on robots.txt alone

CheckedSome operators publish a robots.txt token but never a literal user-agent string. Those bots cannot be impersonated honestly, so they are judged on your rules only — no live request was made in their name.

Found1 of 21: NeticleBot

Worth knowingNothing to do. Their robots.txt verdict is still exact — it is read from your own file — only the server-side half is unknown.

The E. SEO Tools group, bot by bot

Every bot of the E. SEO Tools group, with its two verdicts
Bot server-robots-instructions server-crawl-live-test What stopped it
AhrefsBot Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +97 		HTTP 403 the server (HTTP 403)
AudigentAdBot Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +97 		HTTP 403 the server (HTTP 403)
AwarioRssBot Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +97 		HTTP 403 the server (HTTP 403)
AwarioSmartBot Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +97 		HTTP 403 the server (HTTP 403)
Brandwatch
The vendor's own page says the string varies, so this literal is low confidence. 		Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +97 		HTTP 403 the server (HTTP 403)
Chrome-Lighthouse Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +97 		HTTP 403 the server (HTTP 403)
DataForSeoBot
The crawler behind the data AkOOre's own reports are built on. 		Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +97 		HTTP 403 the server (HTTP 403)
DotBot
A newer DotBot/2.0; +https://dotbot.com/about/ form is also reported but unconfirmed. 		Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +97 		HTTP 403 the server (HTTP 403)
Google Page Speed Insights
alias — same bot as another row
The documented legacy form. Recent PSI requests reportedly drop these tokens for a plain Chrome user-agent. 		Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +97 		HTTP 403 the server (HTTP 403)
Jetslide Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +97 		HTTP 403 the server (HTTP 403)
MJ12bot
Newer v2.0.x builds are also reported, same format. 		Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +97 		HTTP 403 the server (HTTP 403)
Meltwater Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +97 		HTTP 403 the server (HTTP 403)
NeticleBot
The vendor has a bot-info page but quotes no exact header. 		Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +97 		Not probed —
Netvibes Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +97 		HTTP 403 the server (HTTP 403)
Screaming Frog SEO Spider
The version tracks the installed app and changes every release. 		Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +97 		HTTP 403 the server (HTTP 403)
SearchmetricsBot Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +97 		HTTP 403 the server (HTTP 403)
SemrushBot
Older SemrushBot-BA / -SA / -SI variants also exist. 		Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +97 		HTTP 403 the server (HTTP 403)
SemrushBot-OCOB
The Content Toolkit crawler. 		Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +97 		HTTP 403 the server (HTTP 403)
SemrushBotSwa Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +97 		HTTP 403 the server (HTTP 403)
Sidetrade Indexer Bot Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +97 		HTTP 403 the server (HTTP 403)
peer39_crawler
Ad-context classification. The source list names it twice (SEO Tools and Other Agents); it is one bot, listed once. 		Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +97 		HTTP 403 the server (HTTP 403)

Section K / OMeasured group

Scrapers

Guide

This section answers

Which bots of this group can read your site, and what stops the others?

Terms used

Gate 1Gate 2captcha wall

Evidence

Measured Oct. 12, 2026, 2:04 a.m.

Your robots.txt says yes to all 46. Your server says no to 34.

46 bots knock F. Scrapers

Gate 1 Your robots.txt

1 of them have a group of their own in your file, 45 ride on the catch-all group (User-agent: *) and 0 are named by no rule. 46 of them are allowed with exceptions — the closed paths are housekeeping only, no known content page is among them.

Gate 2 Your server, live

ServedRefusedNot probed

1 Read your site CCBot
34 Turned away by the server Arquivo-web-crawler HTTP 403BLEXBot HTTP 403Barkrowler HTTP 403Bytespider HTTP 403Cotoyogi HTTP 403Crawl4AI HTTP 403Diffbot HTTP 403EchoboxBot HTTP 403Factset_spyderbot HTTP 403Firecrawl HTTP 403 + 24 more
11 Judged on the file alone Amazon KendraBravestCrawlspaceEchobot BotJenkersBotPoseidon Research CrawlerSeekrSeekrBothada.newsimediaethics.org + 1 more

Their operator publishes no user-agent string, so no honest knock is possible.

What the measurement means, point by point

Info Server-robots-instructions — what robots.txt permits

CheckedThe first thing every well-behaved bot reads. A Disallow rule on the homepage stops it before it ever requests a page, so nothing further down can compensate. A group that closes some paths but not the homepage is 'allowed with exceptions' — it counts as allowed, and the exceptions are tested against your known content pages below. A bot no rule names is allowed — silence is permission — and shown as 'not indicated'.

Foundall 46 are allowed by robots.txt; 46 of them are allowed with exceptions — the closed paths are housekeeping only, no known content page is among them.

Info Server-crawl-live-test — the live HEAD request

CheckedWhat your server really answers to one HEAD request wearing that bot's name. A firewall, a CDN rule or a bot-protection product can refuse a crawler your robots.txt welcomes — and robots.txt alone can never reveal it.

Found34 forbidden: Arquivo-web-crawler, BLEXBot, Barkrowler, Bytespider, Cotoyogi, Crawl4AI, Diffbot, EchoboxBot and 26 more

Worth knowingRead this one before acting: an ordinary desktop browser was refused too, so your server is not singling out crawlers — it is refusing our test requests wholesale (a rate limit, a geo or IP rule, or a challenge page). Check what your firewall does with unfamiliar visitors before changing any bot rule.

Info Indexing directives in the headers

CheckedThe X-Robots-Tag response header of the answers your server actually served. A noindex here lets the bot in and then tells it to throw the page away — the most expensive way to be invisible, because it looks fine in every other check. Only served answers are read: a wall's or an error page's header is not yours.

Foundno X-Robots-Tag header in any served answer

Info Do the two tests agree?

CheckedWhether the server-robots-instructions and the server-crawl-live-test tell the same story. They often do not: robots.txt is a published promise, the live answer is what actually happens, and only the comparison shows which one to believe.

Found34 bot(s) invited by robots.txt but refused by the server — the same ones the live test above names

Worth knowingA contradiction is worth resolving whichever way you meant it. Invited-but-refused: your firewall overrules your robots.txt — align them, or accept that the file is advertising a welcome the server does not honour. Blocked-but-served: your robots.txt line is a polite request only; a bot that ignores it gets the page, so add a server rule if the block really matters.

Info Judged on robots.txt alone

CheckedSome operators publish a robots.txt token but never a literal user-agent string. Those bots cannot be impersonated honestly, so they are judged on your rules only — no live request was made in their name.

Found11 of 46: Amazon Kendra, Bravest, Crawlspace, Echobot Bot, JenkersBot, Poseidon Research Crawler, Seekr, SeekrBot and 3 more

Worth knowingNothing to do. Their robots.txt verdict is still exact — it is read from your own file — only the server-side half is unknown.

The F. Scrapers group, bot by bot

Every bot of the F. Scrapers group, with its two verdicts
Bot server-robots-instructions server-crawl-live-test What stopped it
Amazon Kendra
AWS publishes the robots.txt token but no literal HTTP header. 		Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +97 		Not probed —
Arquivo-web-crawler Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +97 		HTTP 403 the server (HTTP 403)
BLEXBot Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +97 		HTTP 403 the server (HTTP 403)
Barkrowler Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +97 		HTTP 403 the server (HTTP 403)
Bravest
No published user-agent string. 		Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +97 		Not probed —
Bytespider
ByteDance never published an official documentation page; the Bytespider token is the stable part. 		Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +97 		HTTP 403 the server (HTTP 403)
CCBot
Common Crawl feeds most open models' training sets. 		Allowed with exceptions
by User-agent: ccbot
closed: /admin/, /*/admin/, /manage/ +96 		Redirected —
Cotoyogi Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +97 		HTTP 403 the server (HTTP 403)
Crawl4AI
A library default the caller can override. 		Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +97 		HTTP 403 the server (HTTP 403)
Crawlspace
No published user-agent string. 		Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +97 		Not probed —
Diffbot
Version numbers beyond 0.1 are unverified. 		Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +97 		HTTP 403 the server (HTTP 403)
Echobot Bot
Not to be confused with EchoboxBot above: different company, different bot. 		Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +97 		Not probed —
EchoboxBot
The hash segment is per-deployment. 		Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +97 		HTTP 403 the server (HTTP 403)
Factset_spyderbot Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +97 		HTTP 403 the server (HTTP 403)
Firecrawl Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +97 		HTTP 403 the server (HTTP 403)
FriendlyCrawler Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +97 		HTTP 403 the server (HTTP 403)
ICC-Crawler Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +97 		HTTP 403 the server (HTTP 403)
ISSCyberRiskCrawler Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +97 		HTTP 403 the server (HTTP 403)
ImagesiftBot Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +97 		HTTP 403 the server (HTTP 403)
JenkersBot
No published user-agent string. 		Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +97 		Not probed —
Kangaroo Bot
A differently spelled KangarooBot/1.0; +https://kangaroobot.com (no space) is documented elsewhere -- check which one matches your logs. 		Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +97 		HTTP 403 the server (HTTP 403)
NewsNow Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +97 		HTTP 403 the server (HTTP 403)
NovaAct Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +97 		HTTP 403 the server (HTTP 403)
Poseidon Research Crawler
Only the bare identifier is documented. 		Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +97 		Not probed —
QualifiedBot Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +97 		HTTP 403 the server (HTTP 403)
Scrapy
The library default template; the version tracks the installed Scrapy release. 		Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +97 		HTTP 403 the server (HTTP 403)
SeekportBot Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +97 		HTTP 403 the server (HTTP 403)
Seekr
alias — same bot as another row
No published user-agent string. 		Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +97 		Not probed —
SeekrBot
No published user-agent string. 		Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +97 		Not probed —
TaraGroup Intelligent Bot Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +97 		HTTP 403 the server (HTTP 403)
Timpibot Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +97 		HTTP 403 the server (HTTP 403)
Turnitin Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +97 		HTTP 403 the server (HTTP 403)
VelenPublicWebCrawler Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +97 		HTTP 403 the server (HTTP 403)
Webzio-Extended Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +97 		HTTP 403 the server (HTTP 403)
coccocbot-web Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +97 		HTTP 403 the server (HTTP 403)
crawler4j
A library default, set per deployment by the developer. 		Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +97 		HTTP 403 the server (HTTP 403)
hada.news
No published user-agent string. 		Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +97 		Not probed —
iaskspider Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +97 		HTTP 403 the server (HTTP 403)
imediaethics.org
No published user-agent string. 		Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +97 		Not probed —
imgproxy
A self-hosted image-proxy tool rather than a crawler; the version tracks its release. 		Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +97 		HTTP 403 the server (HTTP 403)
magpie-crawler
A legacy crawler name, distinct from the newer Brandwatch user-agent in SEO Tools. 		Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +97 		HTTP 403 the server (HTTP 403)
netEstate Imprint Crawler
Not to be confused with the separately documented netEstate NE Crawler (+http://www.website-datenbank.de/), an older bot. 		Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +97 		Not probed —
news-please
A library default template, not a fixed signature. 		Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +97 		HTTP 403 the server (HTTP 403)
omgili Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +97 		HTTP 403 the server (HTTP 403)
omgilibot
An older omgilibot/0.3 +http://www.omgili.com/Crawler.html form is also documented. 		Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +97 		HTTP 403 the server (HTTP 403)
yacybot
The bracketed system info varies per peer; yacybot is the stable token. 		Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +97 		HTTP 403 the server (HTTP 403)

Section L / OMeasured group

Cloud Services

Guide

This section answers

Which bots of this group can read your site, and what stops the others?

Terms used

Gate 1Gate 2captcha wall

Evidence

Measured Oct. 12, 2026, 2:04 a.m.

Your robots.txt says yes to all 4. Your server says no to 4.

4 bots knock G. Cloud Services

Gate 1 Your robots.txt

All of them ride on your catch-all group (User-agent: *) — no rule in your file names one of them individually. 4 of them are allowed with exceptions — the closed paths are housekeeping only, no known content page is among them.

Gate 2 Your server, live

4 Turned away by the server AliyunSecBot HTTP 403Amazonbot HTTP 403Amzn-User HTTP 403Google-CloudVertexBot HTTP 403

What the measurement means, point by point

Info Server-robots-instructions — what robots.txt permits

CheckedThe first thing every well-behaved bot reads. A Disallow rule on the homepage stops it before it ever requests a page, so nothing further down can compensate. A group that closes some paths but not the homepage is 'allowed with exceptions' — it counts as allowed, and the exceptions are tested against your known content pages below. A bot no rule names is allowed — silence is permission — and shown as 'not indicated'.

Foundall 4 are allowed by robots.txt; 4 of them are allowed with exceptions — the closed paths are housekeeping only, no known content page is among them.

Info Server-crawl-live-test — the live HEAD request

CheckedWhat your server really answers to one HEAD request wearing that bot's name. A firewall, a CDN rule or a bot-protection product can refuse a crawler your robots.txt welcomes — and robots.txt alone can never reveal it.

Found4 forbidden: AliyunSecBot, Amazonbot, Amzn-User, Google-CloudVertexBot

Worth knowingRead this one before acting: an ordinary desktop browser was refused too, so your server is not singling out crawlers — it is refusing our test requests wholesale (a rate limit, a geo or IP rule, or a challenge page). Check what your firewall does with unfamiliar visitors before changing any bot rule.

Info Do the two tests agree?

CheckedWhether the server-robots-instructions and the server-crawl-live-test tell the same story. They often do not: robots.txt is a published promise, the live answer is what actually happens, and only the comparison shows which one to believe.

Found4 bot(s) invited by robots.txt but refused by the server — the same ones the live test above names

Worth knowingA contradiction is worth resolving whichever way you meant it. Invited-but-refused: your firewall overrules your robots.txt — align them, or accept that the file is advertising a welcome the server does not honour. Blocked-but-served: your robots.txt line is a polite request only; a bot that ignores it gets the page, so add a server rule if the block really matters.

The G. Cloud Services group, bot by bot

Every bot of the G. Cloud Services group, with its two verdicts
Bot server-robots-instructions server-crawl-live-test What stopped it
AliyunSecBot
An unusual format, but the canonical documented instance. 		Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +97 		HTTP 403 the server (HTTP 403)
Amazonbot
Real instances wrap this inside a full desktop or mobile Chrome-style user-agent. 		Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +97 		HTTP 403 the server (HTTP 403)
Amzn-User Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +97 		HTTP 403 the server (HTTP 403)
Google-CloudVertexBot Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +97 		HTTP 403 the server (HTTP 403)

Section M / OMeasured group

Other Agents

Guide

This section answers

Which bots of this group can read your site, and what stops the others?

Terms used

Gate 1Gate 2captcha wall

Evidence

Measured Oct. 12, 2026, 2:04 a.m.

Your robots.txt says yes to all 5. Your server says no to 2.

5 bots knock H. Other Agents

Gate 1 Your robots.txt

All of them ride on your catch-all group (User-agent: *) — no rule in your file names one of them individually. 5 of them are allowed with exceptions — the closed paths are housekeeping only, no known content page is among them.

Gate 2 Your server, live

ServedRefusedNot probed

2 Read your site ia_archiveria_archiver-web.archive.org
2 Turned away by the server TurnitinBot HTTP 403archive.org_bot HTTP 403
1 Judged on the file alone ViennaTinyBot

Their operator publishes no user-agent string, so no honest knock is possible.

What the measurement means, point by point

Info Server-robots-instructions — what robots.txt permits

CheckedThe first thing every well-behaved bot reads. A Disallow rule on the homepage stops it before it ever requests a page, so nothing further down can compensate. A group that closes some paths but not the homepage is 'allowed with exceptions' — it counts as allowed, and the exceptions are tested against your known content pages below. A bot no rule names is allowed — silence is permission — and shown as 'not indicated'.

Foundall 5 are allowed by robots.txt; 5 of them are allowed with exceptions — the closed paths are housekeeping only, no known content page is among them.

Info Server-crawl-live-test — the live HEAD request

CheckedWhat your server really answers to one HEAD request wearing that bot's name. A firewall, a CDN rule or a bot-protection product can refuse a crawler your robots.txt welcomes — and robots.txt alone can never reveal it.

Found2 forbidden: TurnitinBot, archive.org_bot

Worth knowingRead this one before acting: an ordinary desktop browser was refused too, so your server is not singling out crawlers — it is refusing our test requests wholesale (a rate limit, a geo or IP rule, or a challenge page). Check what your firewall does with unfamiliar visitors before changing any bot rule.

Info Indexing directives in the headers

CheckedThe X-Robots-Tag response header of the answers your server actually served. A noindex here lets the bot in and then tells it to throw the page away — the most expensive way to be invisible, because it looks fine in every other check. Only served answers are read: a wall's or an error page's header is not yours.

Foundno X-Robots-Tag header in any served answer

Info Do the two tests agree?

CheckedWhether the server-robots-instructions and the server-crawl-live-test tell the same story. They often do not: robots.txt is a published promise, the live answer is what actually happens, and only the comparison shows which one to believe.

Found2 bot(s) invited by robots.txt but refused by the server — the same ones the live test above names

Worth knowingA contradiction is worth resolving whichever way you meant it. Invited-but-refused: your firewall overrules your robots.txt — align them, or accept that the file is advertising a welcome the server does not honour. Blocked-but-served: your robots.txt line is a polite request only; a bot that ignores it gets the page, so add a server rule if the block really matters.

Info Judged on robots.txt alone

CheckedSome operators publish a robots.txt token but never a literal user-agent string. Those bots cannot be impersonated honestly, so they are judged on your rules only — no live request was made in their name.

Found1 of 5: ViennaTinyBot

Worth knowingNothing to do. Their robots.txt verdict is still exact — it is read from your own file — only the server-side half is unknown.

The H. Other Agents group, bot by bot

Every bot of the H. Other Agents group, with its two verdicts
Bot server-robots-instructions server-crawl-live-test What stopped it
TurnitinBot
A newer TurnitinBot/2.x is reported but not independently confirmed. 		Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +97 		HTTP 403 the server (HTTP 403)
ViennaTinyBot
Only a bare lowercase identifier appears in aggregator directories. 		Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +97 		Not probed —
archive.org_bot
Heritrix-version-qualified variants are also documented. 		Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +97 		HTTP 403 the server (HTTP 403)
ia_archiver
The historical Alexa string. Alexa retired in 2022; the string is kept for legacy matching. 		Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +97 		Redirected —
ia_archiver-web.archive.org
A distinct token used by the Wayback Machine's live-web / Save Page Now fetcher. 		Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +97 		Redirected —

Section N / ORead from the figures

Conclusion

Guide

This section answers

What does the whole picture mean for this site, and what do I change first?

Findings · 2026-10-12

Written Oct. 12, 2026, 2:14 a.m. from the measurements above

Who can read your site

The robots.txt policy for OriginalUniqueArt.com is highly permissive, allowing 100% of the 27 search engine bots and all 33 AI bots to access the site. Within the Google Bots group, 90% are allowed, though Googlebot-Image is not allowed by its own group's rule. While the policy is open, ground-truth evidence shows that Google is actively reading the site, as evidenced by 50 impressions in organic search results over the last 28 days. This confirms that the site is not invisible to search engines despite any probe results. The live server test could not be measured from our network this run and therefore proves nothing about how the site treats any bot.

What is costing you

  • Googlebot-Image (Google Bots group) is blocked by its own rule, which closes the homepage to it.
  • Twitterbot (social bots) is currently stopped by its own robots.txt rule.
  • facebookexternalhit (social bots) is currently stopped by its own robots.txt rule.

What is a choice, not a bug

The current policy allows 100% of SEO tools, scrapers, cloud services, and other agents to access the site, with only housekeeping paths closed. This includes all 33 AI bots, including those that quote and link you in an answer and training crawlers. This high level of permissiveness ensures that your content is available to a wide range of automated agents and AI answer engines, as all AI bots are permitted to access the site's main pages.

Section O / OReference

Words and questions

→
Guide

This section answers

What do the words of this report mean, and what do owners usually ask?

The seven words this report uses

Gate 1 · server-robots-instructions
What your robots.txt says about that bot's name. Costs your server one request in total: the file is fetched once and read for all 157 bots.
Gate 2 · server-crawl-live-test
What your server answers to one live HEAD request wearing that bot's own user-agent. Headers only — no page content travels.
Allowed / Allowed with exceptions / Not allowed / Not indicated
The four robots.txt verdicts. Allowed with exceptions means the homepage is open but the governing group closes some paths — the normal shape of a well-kept file; it counts as allowed, and the closed paths are tested against your known content pages. Not allowed means the homepage itself is closed. Not indicated means no rule names the bot: by the standard, silence is permission.
Served
A normal answer to the live request — that bot gets your page.
Refused · captcha wall
An HTTP refusal (403 and its neighbours), or a success code that led to a challenge page instead of your homepage. Bots do not solve captchas, so the report counts a wall as a refusal and names it.
Rate-limited
HTTP 429: a statement about how fast our probe asked, not about who may crawl you. Excluded from every figure.
Not probed
The operator publishes a robots.txt token but no user-agent string, so no honest knock is possible. Judged on your file alone.

Frequently asked questions

What is the server-robots-instructions test?

It reads your robots.txt — the public rulebook every well-behaved bot fetches before anything else — and looks up each bot's name in it. Four answers are possible: Allowed (the governing group closes nothing), Allowed with exceptions (the homepage is open but some paths are closed — the report lists them and says whether any is a page you publish), Not allowed (the homepage itself is closed), and Not indicated (no rule names it and no catch-all exists — by the standard, silence means permission). This test costs your server one request in total: the file is fetched once and read for all 157 bots.

What is the server-crawl-live-test?

One real HTTP HEAD request to your homepage per bot, sent with that bot's own User-Agent header — the way the bot itself would knock. HEAD asks for the headers only, so no page content travels and the test costs your server almost nothing. The answer is Allowed (a normal response), Forbidden (an HTTP refusal such as 403, or a captcha wall), or Error (a timeout, a network failure or a broken 5xx answer).

What is html-meta-robots — and why is it not in this report?

html-meta-robots is the <meta name="robots" content="noindex"> tag inside a page's HTML: a third place where a site can tell bots what to do, page by page. This report never downloads your HTML (the live test is header-only by design), so it cannot read that tag — but it does read the tag's HTTP twin, the X-Robots-Tag response header, which arrives with every answer: that is the "Indexing directives in the headers" check in each group. The html-meta-robots of each promoted page is checked in the page SEO/GEO reports, which do read the HTML.

Can the two tests contradict each other?

Yes, in both directions, and the contradiction is one of the most useful findings here. Invited but refused: robots.txt allows a bot, yet the server turns its request away — a firewall or bot wall is overruling the file, possibly without anyone having decided that. Blocked but served: robots.txt forbids a bot, yet the server answers anyway — the block is a polite request that only honest bots obey. The "Do the two tests agree?" check in each group lists every case found.

Does a robots.txt block actually stop a bot?

Only if the bot chooses to obey it. robots.txt is a convention, not an enforcement: the major search engines and AI crawlers respect it scrupulously, but nothing technical prevents a scraper from ignoring the file and requesting your pages anyway. If a block really matters to you, it needs a server-side rule too — which is exactly what the server-crawl-live-test measures.

Why might Googlebot show "Forbidden" here while Google still indexes my site?

Because this test sends Googlebot's name from AkOOre's network, not from Google's. Google, Bing and Apple all recommend that sites verify crawlers by IP address or reverse DNS rather than by name — a server doing that correctly refuses anyone who merely claims to be Googlebot while serving the real one normally. That is why the report treats a refusal as a question to investigate (check Search Console or your server logs), and why the control request exists.

What does "Captcha" mean in the live-test column?

The server answered with a success code but sent the bot to a challenge page — a CAPTCHA or a JavaScript interstitial — instead of your homepage. For a crawler that is a refusal with extra steps: bots do not solve captchas, so the page behind the wall is never read. The report counts it as Forbidden and names the challenge, because a naive reading of the 200 status would have called it allowed.

What does "Not probed" mean?

Some bot operators publish a robots.txt token but never a literal user-agent string (Google-Extended and Applebot-Extended, for example, ride along inside other crawlers' requests). Those bots cannot be impersonated honestly — inventing a string would mean inventing a verdict — so they are judged on your robots.txt alone and excluded from the request percentage.

What does "Rate-limited — not measurable" mean?

Your server answered HTTP 429 ("too many requests") — a statement about how fast our probe asked, not about whether that bot is welcome. Counting it as a refusal would blame your site for our pace, so those rows are excluded from every percentage. The probe slows itself down when it sees a 429; re-running later usually measures them cleanly.

Should I block the AI bots?

Split them in two first. Citation crawlers (OAI-SearchBot, ChatGPT-User, Claude-SearchBot, PerplexityBot…) fetch pages so an assistant can quote and link you — blocking them makes you invisible in answers your customers increasingly read. Training crawlers (GPTBot, CCBot, Meta-ExternalAgent…) collect content to train models — blocking them protects your content and costs you no citations. Blocking training while allowing citation is a common, coherent middle ground; the AI Bots group shows exactly which door each bot is.

Add to your ToDo list