You are exploring AkOOre's demo website — read-only, nothing you change is saved.

Crawler access — originaluniqueart.com — generated on Nov. 22, 2026, 11:12 p.m. by AkOOre

OverviewCrawler access · originaluniqueart.com · 16 Nov 2026

Your server is refusing requests wholesale, blocking even normal desktop browsers

This report shows which bots can reach your site and which are being turned away. You have a great opportunity to open your gates by fixing a server rule that is currently blocking many visitors.

Start with section A. Work orders to see the first critical fix. Check your firewall or rate limits to stop these wholesale refusals and let the right traffic in.

Some sections are not available yet.

Introduction written 26 Oct 2026 from this report's data · the data has moved since; it is being rewritten

No
Google
Yes
Bing
No
ChatGPT
No
Perplexity
No
Claude
No
Gemini

In this report

6 sections · 157 bots · 8 groups · 39 checks
originaluniqueart.com · 16 Nov 2026

Section A

Who can read your site

Critical

Work orders

6 critical fixes, 27 in all.

Fix first — critical 6

  1. Server-crawl-live-test — the live HEAD request (D. Google's own robots — this check scores 0/100)
    What to do: Read this one before acting: an ordinary desktop browser was refused too, so your server is not singling out crawlers — it is refusing our test requests wholesale (a rate limit, a geo or IP rule, or a challenge page). Check what your firewall does with unfamiliar visitors before changing any bot rule.
    What we measured on your site: 8 forbidden: Gemini-Deep-Research, Google-InspectionTool, GoogleOther, GoogleOther-Image, GoogleOther-Video, Googlebot-Image, Googlebot-Video, Storebot-Google
    Why it matters: What your server really answers to one HEAD request wearing that bot's name. A firewall, a CDN rule or a bot-protection product can refuse a crawler your robots.txt welcomes — and robots.txt alone can never reveal it.
    Where to see the detail: the D. Google's own robots section shows this check with the data it is based on.
  2. The bots that decide the outcome (C. AI assistants — this check scores 12/100)
    What to do: Fix these before anything else in this report: each one is a whole audience that cannot see the site today. The table below names the exact rule or answer responsible for each.
    What we measured on your site: ChatGPT-User is stopped by its own robots.txt rule (User-agent: chatgpt-user); Claude-SearchBot is stopped by its own robots.txt rule (User-agent: claude-searchbot); ClaudeBot is stopped by its own r…
    Why it matters: Not every bot in this group is worth the same. These are the ones whose verdict changes what your business gets: the indexes people actually search and the assistants they actually ask.
    Where to see the detail: the C. AI assistants section shows this check with the data it is based on.
  3. Server-crawl-live-test — the live HEAD request (B. Search engines — this check scores 15/100)
    What to do: Read this one before acting: an ordinary desktop browser was refused too, so your server is not singling out crawlers — it is refusing our test requests wholesale (a rate limit, a geo or IP rule, or a challenge page). Check what your firewall does with unfamiliar visitors before changing any bot rule.
    What we measured on your site: 17 forbidden: Amzn-SearchBot, Applebot, AspiegelBot, Baiduspider, BingPreview, Googlebot, MicrosoftPreview, MojeekBot and 9 more
    Why it matters: What your server really answers to one HEAD request wearing that bot's name. A firewall, a CDN rule or a bot-protection product can refuse a crawler your robots.txt welcomes — and robots.txt alone can never reveal it.
    Where to see the detail: the B. Search engines section shows this check with the data it is based on.
  4. Server-robots-instructions — what robots.txt permits (D. Google's own robots — this check scores 30/100)
    What to do: Your robots.txt closes the door to GoogleOther, GoogleOther-Image, GoogleOther-Video, Googlebot-Discovery, Googlebot-Image, Googlebot-News, Googlebot-Video. Open the file and look for the User-agent group named in the table below: unless the block is deliberate, delete its Disallow: / line. This is a one-line change with an outsized effect.
    What we measured on your site: 7 of 10 not allowed: GoogleOther, GoogleOther-Image, GoogleOther-Video, Googlebot-Discovery, Googlebot-Image, Googlebot-News, Googlebot-Video; 3 of them are allowed with exceptions — the closed paths…
    Why it matters: The first thing every well-behaved bot reads. A Disallow rule on the homepage stops it before it ever requests a page, so nothing further down can compensate. A group that closes some paths but not the homepage is 'allowed with exceptions' — it counts as allowed, and the exceptions are tested against your known content pages below. A bot no rule names is allowed — silence is permission — and shown as 'not indicated'.
    Where to see the detail: the D. Google's own robots section shows this check with the data it is based on.
  5. The bots that decide the outcome (D. Google's own robots — this check scores 33/100)
    What to do: Fix these before anything else in this report: each one is a whole audience that cannot see the site today. The table below names the exact rule or answer responsible for each.
    What we measured on your site: GoogleOther is stopped by its own robots.txt rule (User-agent: googleother); Googlebot-Image is stopped by its own robots.txt rule (User-agent: googlebot); your server refused every test request, a p…
    Why it matters: Not every bot in this group is worth the same. These are the ones whose verdict changes what your business gets: the indexes people actually search and the assistants they actually ask.
    Where to see the detail: the D. Google's own robots section shows this check with the data it is based on.
  6. Server-robots-instructions — what robots.txt permits (C. AI assistants — this check scores 45/100)
    What to do: Your robots.txt closes the door to ChatGPT Operator, ChatGPT-User, ChatGPT-User/2.0, Claude-SearchBot, Claude-User, Claude-Web, ClaudeBot, GPTBot and 10 more. Open the file and look for the User-agent group named in the table below: unless the block is deliberate, delete its Disallow: / line. This is a one-line change with an outsized effect.
    What we measured on your site: 18 of 33 not allowed: ChatGPT Operator, ChatGPT-User, ChatGPT-User/2.0, Claude-SearchBot, Claude-User, Claude-Web, ClaudeBot, GPTBot and 10 more; 15 of them are allowed with exceptions — the closed p…
    Why it matters: The first thing every well-behaved bot reads. A Disallow rule on the homepage stops it before it ever requests a page, so nothing further down can compensate. A group that closes some paths but not the homepage is 'allowed with exceptions' — it counts as allowed, and the exceptions are tested against your known content pages below. A bot no rule names is allowed — silence is permission — and shown as 'not indicated'.
    Where to see the detail: the C. AI assistants section shows this check with the data it is based on.

Then improve 3

  1. Server-crawl-live-test — the live HEAD request (C. AI assistants — this check scores 58/100)
    What to do: Read this one before acting: an ordinary desktop browser was refused too, so your server is not singling out crawlers — it is refusing our test requests wholesale (a rate limit, a geo or IP rule, or a challenge page). Check what your firewall does with unfamiliar visitors before changing any bot rule.
    What we measured on your site: 10 forbidden: AI2Bot, Ai2Bot-Dolma, DeepSeekBot, DigitalOceanGenAICrawler, DuckAssistBot, LINER Bot, PanguBot, SBIntuitionsBot and 2 more
    Why it matters: What your server really answers to one HEAD request wearing that bot's name. A firewall, a CDN rule or a bot-protection product can refuse a crawler your robots.txt welcomes — and robots.txt alone can never reveal it.
    Where to see the detail: the C. AI assistants section shows this check with the data it is based on.
  2. Server-crawl-live-test — the live HEAD request (E. Social Bots — this check scores 64/100)
    What to do: Read this one before acting: an ordinary desktop browser was refused too, so your server is not singling out crawlers — it is refusing our test requests wholesale (a rate limit, a geo or IP rule, or a challenge page). Check what your firewall does with unfamiliar visitors before changing any bot rule.
    What we measured on your site: 4 forbidden: Meta-ExternalAds, Meta-ExternalFetcher, Meta-WebIndexer, Quora-Bot
    Why it matters: What your server really answers to one HEAD request wearing that bot's name. A firewall, a CDN rule or a bot-protection product can refuse a crawler your robots.txt welcomes — and robots.txt alone can never reveal it.
    Where to see the detail: the E. Social Bots section shows this check with the data it is based on.
  3. Server-robots-instructions — what robots.txt permits (E. Social Bots — this check scores 73/100)
    What to do: Your robots.txt closes the door to FacebookBot, LinkedInBot, Meta-ExternalAgent. Open the file and look for the User-agent group named in the table below: unless the block is deliberate, delete its Disallow: / line. This is a one-line change with an outsized effect.
    What we measured on your site: 3 of 11 not allowed: FacebookBot, LinkedInBot, Meta-ExternalAgent; 8 of them are allowed with exceptions — the closed paths are housekeeping only, no known content page is among them.
    Why it matters: The first thing every well-behaved bot reads. A Disallow rule on the homepage stops it before it ever requests a page, so nothing further down can compensate. A group that closes some paths but not the homepage is 'allowed with exceptions' — it counts as allowed, and the exceptions are tested against your known content pages below. A bot no rule names is allowed — silence is permission — and shown as 'not indicated'.
    Where to see the detail: the E. Social Bots section shows this check with the data it is based on.

Worth knowing 18

  1. Do the two tests agree? (B. Search engines)
    What to do with it: A contradiction is worth resolving whichever way you meant it. Invited-but-refused: your firewall overrules your robots.txt — align them, or accept that the file is advertising a welcome the server does not honour. Blocked-but-served: your robots.txt line is a polite request only; a bot that ignores it gets the page, so add a server rule if the block really matters.
    What the data says: 16 bot(s) invited by robots.txt but refused by the server: Amzn-SearchBot, Applebot, AspiegelBot, Baiduspider, BingPreview, MicrosoftPreview, MojeekBot, PetalBot and 8 more
    Why it matters: Whether the server-robots-instructions and the server-crawl-live-test tell the same story. They often do not: robots.txt is a published promise, the live answer is what actually happens, and only the comparison shows which one to believe.
    Where to see the detail: the B. Search engines section shows this check with the data it is based on.
  2. Judged on robots.txt alone (B. Search engines)
    What to do with it: Nothing to do. Their robots.txt verdict is still exact — it is read from your own file — only the server-side half is unknown.
    What the data says: 7 of 27: Applebot-Extended, BingVideoPreview, Yahoo-Blogs, YahooSeeker, Yandex, YandexAdditionalBot, baidu
    Why it matters: Some operators publish a robots.txt token but never a literal user-agent string. Those bots cannot be impersonated honestly, so they are judged on your rules only — no live request was made in their name.
    Where to see the detail: the B. Search engines section shows this check with the data it is based on.
  3. Do the two tests agree? (C. AI assistants)
    What to do with it: A contradiction is worth resolving whichever way you meant it. Invited-but-refused: your firewall overrules your robots.txt — align them, or accept that the file is advertising a welcome the server does not honour. Blocked-but-served: your robots.txt line is a polite request only; a bot that ignores it gets the page, so add a server rule if the block really matters.
    What the data says: 10 bot(s) invited by robots.txt but refused by the server — the same ones the live test above names; 13 bot(s) blocked in robots.txt but served anyway: ChatGPT-User, Claude-SearchBot, Claude-User, Cl…
    Why it matters: Whether the server-robots-instructions and the server-crawl-live-test tell the same story. They often do not: robots.txt is a published promise, the live answer is what actually happens, and only the comparison shows which one to believe.
    Where to see the detail: the C. AI assistants section shows this check with the data it is based on.
  4. Judged on robots.txt alone (C. AI assistants)
    What to do with it: Nothing to do. Their robots.txt verdict is still exact — it is read from your own file — only the server-side half is unknown.
    What the data says: 9 of 33: ChatGPT Operator, ChatGPT-User/2.0, Google-Extended, Grok, GrokBot, MistralAI-User, MyCentralAIScraperBot, Perplexity-User and 1 more
    Why it matters: Some operators publish a robots.txt token but never a literal user-agent string. Those bots cannot be impersonated honestly, so they are judged on your rules only — no live request was made in their name.
    Where to see the detail: the C. AI assistants section shows this check with the data it is based on.
  5. Do the two tests agree? (D. Google's own robots)
    What to do with it: A contradiction is worth resolving whichever way you meant it. Invited-but-refused: your firewall overrules your robots.txt — align them, or accept that the file is advertising a welcome the server does not honour. Blocked-but-served: your robots.txt line is a polite request only; a bot that ignores it gets the page, so add a server rule if the block really matters.
    What the data says: 3 bot(s) invited by robots.txt but refused by the server: Gemini-Deep-Research, Google-InspectionTool, Storebot-Google
    Why it matters: Whether the server-robots-instructions and the server-crawl-live-test tell the same story. They often do not: robots.txt is a published promise, the live answer is what actually happens, and only the comparison shows which one to believe.
    Where to see the detail: the D. Google's own robots section shows this check with the data it is based on.
  6. Judged on robots.txt alone (D. Google's own robots)
    What to do with it: Nothing to do. Their robots.txt verdict is still exact — it is read from your own file — only the server-side half is unknown.
    What the data says: 2 of 10: Googlebot-Discovery, Googlebot-News
    Why it matters: Some operators publish a robots.txt token but never a literal user-agent string. Those bots cannot be impersonated honestly, so they are judged on your rules only — no live request was made in their name.
    Where to see the detail: the D. Google's own robots section shows this check with the data it is based on.
  7. Do the two tests agree? (E. Social Bots)
    What to do with it: A contradiction is worth resolving whichever way you meant it. Invited-but-refused: your firewall overrules your robots.txt — align them, or accept that the file is advertising a welcome the server does not honour. Blocked-but-served: your robots.txt line is a polite request only; a bot that ignores it gets the page, so add a server rule if the block really matters.
    What the data says: 4 bot(s) invited by robots.txt but refused by the server — the same ones the live test above names; 3 bot(s) blocked in robots.txt but served anyway: FacebookBot, LinkedInBot, Meta-ExternalAgent
    Why it matters: Whether the server-robots-instructions and the server-crawl-live-test tell the same story. They often do not: robots.txt is a published promise, the live answer is what actually happens, and only the comparison shows which one to believe.
    Where to see the detail: the E. Social Bots section shows this check with the data it is based on.
  8. Server-crawl-live-test — the live HEAD request (E. SEO Tools)
    What to do with it: Read this one before acting: an ordinary desktop browser was refused too, so your server is not singling out crawlers — it is refusing our test requests wholesale (a rate limit, a geo or IP rule, or a challenge page). Check what your firewall does with unfamiliar visitors before changing any bot rule.
    What the data says: 20 forbidden: AhrefsBot, AudigentAdBot, AwarioRssBot, AwarioSmartBot, Brandwatch, Chrome-Lighthouse, DataForSeoBot, DotBot and 12 more
    Why it matters: What your server really answers to one HEAD request wearing that bot's name. A firewall, a CDN rule or a bot-protection product can refuse a crawler your robots.txt welcomes — and robots.txt alone can never reveal it.
    Where to see the detail: the E. SEO Tools section shows this check with the data it is based on.
  9. Do the two tests agree? (E. SEO Tools)
    What to do with it: A contradiction is worth resolving whichever way you meant it. Invited-but-refused: your firewall overrules your robots.txt — align them, or accept that the file is advertising a welcome the server does not honour. Blocked-but-served: your robots.txt line is a polite request only; a bot that ignores it gets the page, so add a server rule if the block really matters.
    What the data says: 20 bot(s) invited by robots.txt but refused by the server — the same ones the live test above names
    Why it matters: Whether the server-robots-instructions and the server-crawl-live-test tell the same story. They often do not: robots.txt is a published promise, the live answer is what actually happens, and only the comparison shows which one to believe.
    Where to see the detail: the E. SEO Tools section shows this check with the data it is based on.
  10. Judged on robots.txt alone (E. SEO Tools)
    What to do with it: Nothing to do. Their robots.txt verdict is still exact — it is read from your own file — only the server-side half is unknown.
    What the data says: 1 of 21: NeticleBot
    Why it matters: Some operators publish a robots.txt token but never a literal user-agent string. Those bots cannot be impersonated honestly, so they are judged on your rules only — no live request was made in their name.
    Where to see the detail: the E. SEO Tools section shows this check with the data it is based on.
  11. Server-crawl-live-test — the live HEAD request (E. Scrapers)
    What to do with it: Read this one before acting: an ordinary desktop browser was refused too, so your server is not singling out crawlers — it is refusing our test requests wholesale (a rate limit, a geo or IP rule, or a challenge page). Check what your firewall does with unfamiliar visitors before changing any bot rule.
    What the data says: 34 forbidden: Arquivo-web-crawler, BLEXBot, Bytespider, CCBot, Cotoyogi, Crawl4AI, Diffbot, EchoboxBot and 26 more
    Why it matters: What your server really answers to one HEAD request wearing that bot's name. A firewall, a CDN rule or a bot-protection product can refuse a crawler your robots.txt welcomes — and robots.txt alone can never reveal it.
    Where to see the detail: the E. Scrapers section shows this check with the data it is based on.
  12. Do the two tests agree? (E. Scrapers)
    What to do with it: A contradiction is worth resolving whichever way you meant it. Invited-but-refused: your firewall overrules your robots.txt — align them, or accept that the file is advertising a welcome the server does not honour. Blocked-but-served: your robots.txt line is a polite request only; a bot that ignores it gets the page, so add a server rule if the block really matters.
    What the data says: 34 bot(s) invited by robots.txt but refused by the server — the same ones the live test above names
    Why it matters: Whether the server-robots-instructions and the server-crawl-live-test tell the same story. They often do not: robots.txt is a published promise, the live answer is what actually happens, and only the comparison shows which one to believe.
    Where to see the detail: the E. Scrapers section shows this check with the data it is based on.
  13. Judged on robots.txt alone (E. Scrapers)
    What to do with it: Nothing to do. Their robots.txt verdict is still exact — it is read from your own file — only the server-side half is unknown.
    What the data says: 11 of 46: Amazon Kendra, Bravest, Crawlspace, Echobot Bot, JenkersBot, Poseidon Research Crawler, Seekr, SeekrBot and 3 more
    Why it matters: Some operators publish a robots.txt token but never a literal user-agent string. Those bots cannot be impersonated honestly, so they are judged on your rules only — no live request was made in their name.
    Where to see the detail: the E. Scrapers section shows this check with the data it is based on.
  14. Server-crawl-live-test — the live HEAD request (E. Cloud Services)
    What to do with it: Read this one before acting: an ordinary desktop browser was refused too, so your server is not singling out crawlers — it is refusing our test requests wholesale (a rate limit, a geo or IP rule, or a challenge page). Check what your firewall does with unfamiliar visitors before changing any bot rule.
    What the data says: 4 forbidden: AliyunSecBot, Amazonbot, Amzn-User, Google-CloudVertexBot
    Why it matters: What your server really answers to one HEAD request wearing that bot's name. A firewall, a CDN rule or a bot-protection product can refuse a crawler your robots.txt welcomes — and robots.txt alone can never reveal it.
    Where to see the detail: the E. Cloud Services section shows this check with the data it is based on.
  15. Do the two tests agree? (E. Cloud Services)
    What to do with it: A contradiction is worth resolving whichever way you meant it. Invited-but-refused: your firewall overrules your robots.txt — align them, or accept that the file is advertising a welcome the server does not honour. Blocked-but-served: your robots.txt line is a polite request only; a bot that ignores it gets the page, so add a server rule if the block really matters.
    What the data says: 4 bot(s) invited by robots.txt but refused by the server — the same ones the live test above names
    Why it matters: Whether the server-robots-instructions and the server-crawl-live-test tell the same story. They often do not: robots.txt is a published promise, the live answer is what actually happens, and only the comparison shows which one to believe.
    Where to see the detail: the E. Cloud Services section shows this check with the data it is based on.
  16. Server-crawl-live-test — the live HEAD request (E. Other Agents)
    What to do with it: Read this one before acting: an ordinary desktop browser was refused too, so your server is not singling out crawlers — it is refusing our test requests wholesale (a rate limit, a geo or IP rule, or a challenge page). Check what your firewall does with unfamiliar visitors before changing any bot rule.
    What the data says: 1 forbidden: TurnitinBot
    Why it matters: What your server really answers to one HEAD request wearing that bot's name. A firewall, a CDN rule or a bot-protection product can refuse a crawler your robots.txt welcomes — and robots.txt alone can never reveal it.
    Where to see the detail: the E. Other Agents section shows this check with the data it is based on.
  17. Do the two tests agree? (E. Other Agents)
    What to do with it: A contradiction is worth resolving whichever way you meant it. Invited-but-refused: your firewall overrules your robots.txt — align them, or accept that the file is advertising a welcome the server does not honour. Blocked-but-served: your robots.txt line is a polite request only; a bot that ignores it gets the page, so add a server rule if the block really matters.
    What the data says: 1 bot(s) invited by robots.txt but refused by the server — the same ones the live test above names
    Why it matters: Whether the server-robots-instructions and the server-crawl-live-test tell the same story. They often do not: robots.txt is a published promise, the live answer is what actually happens, and only the comparison shows which one to believe.
    Where to see the detail: the E. Other Agents section shows this check with the data it is based on.
  18. Judged on robots.txt alone (E. Other Agents)
    What to do with it: Nothing to do. Their robots.txt verdict is still exact — it is read from your own file — only the server-side half is unknown.
    What the data says: 1 of 5: ViennaTinyBot
    Why it matters: Some operators publish a robots.txt token but never a literal user-agent string. Those bots cannot be impersonated honestly, so they are judged on your rules only — no live request was made in their name.
    Where to see the detail: the E. Other Agents section shows this check with the data it is based on.

Who can actually read your site

98 of 157 bots turned away.

157 bots tested 8 groups 126 probed live with their own user-agent last tested Nov. 16, 2026, 2:03 a.m.

Your robots.txt lets 127 of 157 in. Your server says no to 98.

157 bots knock originaluniqueart.com

Gate 1 Your robots.txt

Allowed with exceptionsNot allowed

Your file names and blocks ChatGPT Operator, ChatGPT-User, ChatGPT-User/2.0, Claude-SearchBot, Claude-User, Claude-Web, ClaudeBot, GPTBot and 22 more. 127 of them are allowed with exceptions — the closed paths are housekeeping only, no known content page is among them.

Gate 2 Your server, live

ServedCaptcha wallRefusedNot probed

28 Read your site ChatGPT-UserClaude-CodeClaude-SearchBotClaude-UserClaude-WebClaudeBotGPTBotMistralAI-User/1.0OAI-SearchBotPerplexity-User/1.0 + 18 more
98 Turned away by the server AI2Bot HTTP 403Ai2Bot-Dolma HTTP 403DeepSeekBot HTTP 403DigitalOceanGenAICrawler HTTP 403DuckAssistBot HTTP 403LINER Bot HTTP 403PanguBot HTTP 403SBIntuitionsBot HTTP 403cohere-training-data-crawler HTTP 403img2dataset HTTP 403 + 88 more
31 Judged on the file alone ChatGPT OperatorChatGPT-User/2.0Google-ExtendedGrokGrokBotMistralAI-UserMyCentralAIScraperBotPerplexity-Userquillbot.comGooglebot-Discovery + 21 more

Their operator publishes no user-agent string, so no honest knock is possible.

Control request: a plain browser

HTTP 403 — a plain browser was refused too

Sent as Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/140.0.0.0 Safari/537.36

Let them in: paste this into your robots.txt

Add these lines to the robots.txt file at the root of your website. A robot follows the group that names it and ignores the catch-all group (User-agent: *), so copy under each new group the Disallow lines of the private paths it should still skip. If your file already has a group for one of these robots, delete that group's Disallow: / line instead.

User-agent: ChatGPT-User
Allow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: Claude-User
Allow: /

User-agent: ClaudeBot
Allow: /

User-agent: GPTBot
Allow: /

User-agent: Google-Extended
Allow: /

User-agent: OAI-SearchBot
Allow: /

User-agent: Perplexity-User
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: Googlebot
Allow: /

The figures and the access standing

Critical

49/100, 6 critical checks.

49/100 Critical

05080100
Critical
6
Fix soon
3
Good
3
127/157 Allowed in robots.txt
28/126 Requests allowed
25/27 Search engines welcome
15/33 AI answer engines welcome
98 Requests forbidden

Every group, and which door it is behind

Lowest scored group: Google Bots, 21%.

Every bot group, with how many bots got each outcome
GroupScore Blocked by robots.txtServedCaptcha wallRefusedNot probed
Scored — being shut out here costs you
Search Engines 64% Fix soon 2 3 1 16 7
AI Bots 38% Critical 18 14 · 10 9
Google Bots 21% Critical 7 · 5 3 2
Social Bots 72% Fix soon 3 7 · 4 ·
Measured, never marked wrong
SEO Tools not scored · · · 20 1
Scrapers not scored · 1 1 33 11
Cloud Services not scored · · · 4 ·
Other Agents not scored · 3 · 1 1

The posture we measured

108 contradictions between file and server.

Your posture

Hard wall

robots.txt welcomes the bots, but the server turns most requests away (a firewall, CDN rule or bot challenge). The published rulebook and the door policy disagree — the door wins.

What it buys you

  • Strong, enforced protection: bots that ignore robots.txt are stopped anyway.
  • Legitimate if crawler verification by IP or reverse DNS lets the real search engines through a door that name-claimers cannot use.

What it costs you

  • If the wall does NOT allowlist real crawlers by IP, search engines and AI assistants are locked out with everyone else — invisibility, not protection.
  • The welcoming robots.txt is misleading — it advertises an access the server does not grant.

Where your two rulebooks contradict each other

92 invited, then refused

robots.txt welcomes AI2Bot, Ai2Bot-Dolma, DeepSeekBot, DigitalOceanGenAICrawler, DuckAssistBot, LINER Bot and 86 more, but the server turns them away. The server wins — your file is advertising a welcome it does not honour.

16 blocked, then served

robots.txt forbids ChatGPT-User, Claude-SearchBot, Claude-User, Claude-Web, ClaudeBot, GPTBot and 10 more, yet the server serves them anyway: the block only holds against bots polite enough to honour it.

Section B · Scored group

Search engines

Fix soon

3 of 27 bots read your site.

Measured Nov. 16, 2026, 2:03 a.m.

2/3 scored points pass. Weakest point: Server-crawl-live-test — the live HEAD request (15%).

Your robots.txt lets 25 of 27 in. Your server says no to 17.

27 bots knock Search engines

Gate 1 Your robots.txt

Allowed with exceptionsNot allowed

Your file names and blocks Applebot-Extended, Googlebot. 25 of them are allowed with exceptions — the closed paths are housekeeping only, no known content page is among them.

Gate 2 Your server, live

ServedCaptcha wallRefusedNot probed

3 Read your site BingbotDuckDuckBotadidxbot
17 Turned away by the server Amzn-SearchBot HTTP 403Applebot HTTP 403AspiegelBot HTTP 403Baiduspider HTTP 403BingPreview HTTP 403MicrosoftPreview HTTP 403MojeekBot HTTP 403PetalBot HTTP 403SeznamBot HTTP 403SeznamHomepageCrawler HTTP 403 + 7 more
7 Judged on the file alone Applebot-ExtendedBingVideoPreviewYahoo-BlogsYahooSeekerYandexYandexAdditionalBotbaidu

Their operator publishes no user-agent string, so no honest knock is possible.

What the measurement means, point by point

Good Server-robots-instructions — what robots.txt permits 93%

CheckedThe first thing every well-behaved bot reads. A Disallow rule on the homepage stops it before it ever requests a page, so nothing further down can compensate. A group that closes some paths but not the homepage is 'allowed with exceptions' — it counts as allowed, and the exceptions are tested against your known content pages below. A bot no rule names is allowed — silence is permission — and shown as 'not indicated'.

Found2 of 27 not allowed: Applebot-Extended, Googlebot; 25 of them are allowed with exceptions — the closed paths are housekeeping only, no known content page is among them.

Critical Server-crawl-live-test — the live HEAD request 15%

CheckedWhat your server really answers to one HEAD request wearing that bot's name. A firewall, a CDN rule or a bot-protection product can refuse a crawler your robots.txt welcomes — and robots.txt alone can never reveal it.

Found17 forbidden: Amzn-SearchBot, Applebot, AspiegelBot, Baiduspider, BingPreview, Googlebot, MicrosoftPreview, MojeekBot and 9 more

Do thisRead this one before acting: an ordinary desktop browser was refused too, so your server is not singling out crawlers — it is refusing our test requests wholesale (a rate limit, a geo or IP rule, or a challenge page). Check what your firewall does with unfamiliar visitors before changing any bot rule.

Worth knowing Indexing directives in the headers

CheckedThe X-Robots-Tag response header of the answers your server actually served. A noindex here lets the bot in and then tells it to throw the page away — the most expensive way to be invisible, because it looks fine in every other check. Only served answers are read: a wall's or an error page's header is not yours.

Foundno X-Robots-Tag header in any served answer

Worth knowing Do the two tests agree?

CheckedWhether the server-robots-instructions and the server-crawl-live-test tell the same story. They often do not: robots.txt is a published promise, the live answer is what actually happens, and only the comparison shows which one to believe.

Found16 bot(s) invited by robots.txt but refused by the server: Amzn-SearchBot, Applebot, AspiegelBot, Baiduspider, BingPreview, MicrosoftPreview, MojeekBot, PetalBot and 8 more

Worth knowingA contradiction is worth resolving whichever way you meant it. Invited-but-refused: your firewall overrules your robots.txt — align them, or accept that the file is advertising a welcome the server does not honour. Blocked-but-served: your robots.txt line is a polite request only; a bot that ignores it gets the page, so add a server rule if the block really matters.

Good The bots that decide the outcome 83%

CheckedNot every bot in this group is worth the same. These are the ones whose verdict changes what your business gets: the indexes people actually search and the assistants they actually ask.

FoundGooglebot is stopped by its own robots.txt rule (User-agent: googlebot); your server refused every test request, a plain browser's included, so its answer to Applebot, Baiduspider, YandexBot is not counted here

Worth knowing Judged on robots.txt alone

CheckedSome operators publish a robots.txt token but never a literal user-agent string. Those bots cannot be impersonated honestly, so they are judged on your rules only — no live request was made in their name.

Found7 of 27: Applebot-Extended, BingVideoPreview, Yahoo-Blogs, YahooSeeker, Yandex, YandexAdditionalBot, baidu

Worth knowingNothing to do. Their robots.txt verdict is still exact — it is read from your own file — only the server-side half is unknown.

The Search engines group, bot by bot

  1. ✓

    adidxbot

    robots.txt: Allowed with exceptions · your server: Redirected

  2. ✗

    Amzn-SearchBot

    robots.txt: Allowed with exceptions · your server: HTTP 403

  3. ✗

    Applebot

    robots.txt: Allowed with exceptions · your server: HTTP 403

  4. !

    Applebot-Extended

    robots.txt: Not allowed · your server: Not probed · its own robots.txt rule (User-agent: applebot-extended)

  5. ✗

    AspiegelBot

    robots.txt: Allowed with exceptions · your server: HTTP 403

  6. !

    baidu

    robots.txt: Allowed with exceptions · your server: Not probed

  7. ✗

    Baiduspider

    robots.txt: Allowed with exceptions · your server: HTTP 403

  8. ✓

    Bingbot

    robots.txt: Allowed with exceptions · your server: Redirected

  9. ✗

    BingPreview

    robots.txt: Allowed with exceptions · your server: HTTP 403

  10. !

    BingVideoPreview

    robots.txt: Allowed with exceptions · your server: Not probed

  11. ✓

    DuckDuckBot

    robots.txt: Allowed with exceptions · your server: Redirected

  12. ✗

    Googlebot

    robots.txt: Not allowed · your server: Captcha · its own robots.txt rule (User-agent: googlebot)

  13. ✗

    MicrosoftPreview

    robots.txt: Allowed with exceptions · your server: HTTP 403

  14. ✗

    MojeekBot

    robots.txt: Allowed with exceptions · your server: HTTP 403

  15. ✗

    PetalBot

    robots.txt: Allowed with exceptions · your server: HTTP 403

  16. ✗

    SeznamBot

    robots.txt: Allowed with exceptions · your server: HTTP 403

  17. ✗

    SeznamHomepageCrawler

    robots.txt: Allowed with exceptions · your server: HTTP 403

  18. ✗

    Slurp

    robots.txt: Allowed with exceptions · your server: HTTP 403

  19. ✗

    Teoma

    robots.txt: Allowed with exceptions · your server: HTTP 403

  20. !

    Yahoo-Blogs

    robots.txt: Allowed with exceptions · your server: Not probed

  21. ✗

    Yahoo-FeedSeeker

    robots.txt: Allowed with exceptions · your server: HTTP 403

  22. ✗

    Yahoo-MMCrawler

    robots.txt: Allowed with exceptions · your server: HTTP 403

  23. !

    YahooSeeker

    robots.txt: Allowed with exceptions · your server: Not probed

  24. !

    Yandex

    robots.txt: Allowed with exceptions · your server: Not probed

  25. ✗

    YandexAdditional

    robots.txt: Allowed with exceptions · your server: HTTP 403

  26. !

    YandexAdditionalBot

    robots.txt: Allowed with exceptions · your server: Not probed

  27. ✗

    YandexBot

    robots.txt: Allowed with exceptions · your server: HTTP 403

See the roster table
Every bot of the Search engines group, with its two verdicts
Bot Gate 1 · your robots.txt Gate 2 · your server, live What stopped it
adidxbot
Bing's ad-landing-page quality crawler.
Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +99
Redirected —
Amzn-SearchBot
Feeds Amazon, Alexa and Rufus search.
Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +99
HTTP 403 the server (HTTP 403)
Applebot
The device, OS and WebKit fields vary between Apple's crawlers.
Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +99
HTTP 403 the server (HTTP 403)
Applebot-Extended
A robots.txt token with no distinct user-agent: it governs whether Apple Intelligence may train on content Applebot already fetched.
Not allowed
by User-agent: applebot-extended
Not probed its own robots.txt rule (User-agent: applebot-extended)
AspiegelBot
alias — same bot as another row
The legacy name, superseded by PetalBot.
Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +99
HTTP 403 the server (HTTP 403)
baidu
alias — same bot as another row
The same bot as Baiduspider, not a separate name or string.
Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +99
Not probed —
Baiduspider Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +99
HTTP 403 the server (HTTP 403)
Bingbot Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +99
Redirected —
BingPreview Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +99
HTTP 403 the server (HTTP 403)
BingVideoPreview
Bing confirms this bot exists but publishes no literal user-agent string, so only its robots.txt token is checked.
Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +99
Not probed —
DuckDuckBot Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +99
Redirected —
Googlebot
The mobile variant wraps a Chrome/Android string around the same (compatible; Googlebot/2.1; ...) suffix.
Not allowed
by User-agent: googlebot
Captcha its own robots.txt rule (User-agent: googlebot)
MicrosoftPreview Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +99
HTTP 403 the server (HTTP 403)
MojeekBot
The version number increments over time (0.2 to 0.11+ seen).
Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +99
HTTP 403 the server (HTTP 403)
PetalBot
Huawei Petal Search. The mobile variant prefixes Android and AppleWebKit fields.
Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +99
HTTP 403 the server (HTTP 403)
SeznamBot Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +99
HTTP 403 the server (HTTP 403)
SeznamHomepageCrawler Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +99
HTTP 403 the server (HTTP 403)
Slurp Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +99
HTTP 403 the server (HTTP 403)
Teoma Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +99
HTTP 403 the server (HTTP 403)
Yahoo-Blogs
Legacy, listed only by unofficial aggregators.
Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +99
Not probed —
Yahoo-FeedSeeker
Historical form, not verified as current.
Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +99
HTTP 403 the server (HTTP 403)
Yahoo-MMCrawler
Historical form, not verified as current.
Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +99
HTTP 403 the server (HTTP 403)
YahooSeeker
Legacy: Yahoo Search is Bing-powered now and no current documentation publishes a string.
Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +99
Not probed —
Yandex
alias — same bot as another row
A robots.txt control token matching any Yandex* bot, not a distinct request user-agent.
Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +99
Not probed —
YandexAdditional
Mostly a robots.txt control token; this string comes from secondary sources, not from Yandex's own documentation.
Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +99
HTTP 403 the server (HTTP 403)
YandexAdditionalBot
alias — same bot as another row
Same bot and same token as YandexAdditional.
Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +99
Not probed —
YandexBot Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +99
HTTP 403 the server (HTTP 403)

Section C · Scored group

AI assistants

Critical

14 of 33 bots read your site.

Measured Nov. 16, 2026, 2:03 a.m.

0/3 scored points pass. Weakest point: The bots that decide the outcome (12%).

Your robots.txt lets 15 of 33 in. Your server says no to 10.

33 bots knock AI assistants

Gate 1 Your robots.txt

Allowed with exceptionsNot allowed

Your file names and blocks ChatGPT Operator, ChatGPT-User, ChatGPT-User/2.0, Claude-SearchBot, Claude-User, Claude-Web, ClaudeBot, GPTBot and 10 more. 15 of them are allowed with exceptions — the closed paths are housekeeping only, no known content page is among them.

Gate 2 Your server, live

ServedRefusedNot probed

14 Read your site ChatGPT-UserClaude-CodeClaude-SearchBotClaude-UserClaude-WebClaudeBotGPTBotMistralAI-User/1.0OAI-SearchBotPerplexity-User/1.0 + 4 more
10 Turned away by the server AI2Bot HTTP 403Ai2Bot-Dolma HTTP 403DeepSeekBot HTTP 403DigitalOceanGenAICrawler HTTP 403DuckAssistBot HTTP 403LINER Bot HTTP 403PanguBot HTTP 403SBIntuitionsBot HTTP 403cohere-training-data-crawler HTTP 403img2dataset HTTP 403
9 Judged on the file alone ChatGPT OperatorChatGPT-User/2.0Google-ExtendedGrokGrokBotMistralAI-UserMyCentralAIScraperBotPerplexity-Userquillbot.com

Their operator publishes no user-agent string, so no honest knock is possible.

What the measurement means, point by point

Critical Server-robots-instructions — what robots.txt permits 45%

CheckedThe first thing every well-behaved bot reads. A Disallow rule on the homepage stops it before it ever requests a page, so nothing further down can compensate. A group that closes some paths but not the homepage is 'allowed with exceptions' — it counts as allowed, and the exceptions are tested against your known content pages below. A bot no rule names is allowed — silence is permission — and shown as 'not indicated'.

Found18 of 33 not allowed: ChatGPT Operator, ChatGPT-User, ChatGPT-User/2.0, Claude-SearchBot, Claude-User, Claude-Web, ClaudeBot, GPTBot and 10 more; 15 of them are allowed with exceptions — the closed paths are housekeeping only, no known content page is among them.

Do thisYour robots.txt closes the door to ChatGPT Operator, ChatGPT-User, ChatGPT-User/2.0, Claude-SearchBot, Claude-User, Claude-Web, ClaudeBot, GPTBot and 10 more. Open the file and look for the User-agent group named in the table below: unless the block is deliberate, delete its Disallow: / line. This is a one-line change with an outsized effect.

Fix soon Server-crawl-live-test — the live HEAD request 58%

CheckedWhat your server really answers to one HEAD request wearing that bot's name. A firewall, a CDN rule or a bot-protection product can refuse a crawler your robots.txt welcomes — and robots.txt alone can never reveal it.

Found10 forbidden: AI2Bot, Ai2Bot-Dolma, DeepSeekBot, DigitalOceanGenAICrawler, DuckAssistBot, LINER Bot, PanguBot, SBIntuitionsBot and 2 more

Do thisRead this one before acting: an ordinary desktop browser was refused too, so your server is not singling out crawlers — it is refusing our test requests wholesale (a rate limit, a geo or IP rule, or a challenge page). Check what your firewall does with unfamiliar visitors before changing any bot rule.

Worth knowing Indexing directives in the headers

CheckedThe X-Robots-Tag response header of the answers your server actually served. A noindex here lets the bot in and then tells it to throw the page away — the most expensive way to be invisible, because it looks fine in every other check. Only served answers are read: a wall's or an error page's header is not yours.

Foundno X-Robots-Tag header in any served answer

Worth knowing Do the two tests agree?

CheckedWhether the server-robots-instructions and the server-crawl-live-test tell the same story. They often do not: robots.txt is a published promise, the live answer is what actually happens, and only the comparison shows which one to believe.

Found10 bot(s) invited by robots.txt but refused by the server — the same ones the live test above names; 13 bot(s) blocked in robots.txt but served anyway: ChatGPT-User, Claude-SearchBot, Claude-User, Claude-Web, ClaudeBot, GPTBot, MistralAI-User/1.0, OAI-SearchBot and 5 more

Worth knowingA contradiction is worth resolving whichever way you meant it. Invited-but-refused: your firewall overrules your robots.txt — align them, or accept that the file is advertising a welcome the server does not honour. Blocked-but-served: your robots.txt line is a polite request only; a bot that ignores it gets the page, so add a server rule if the block really matters.

Critical The bots that decide the outcome 12%

CheckedNot every bot in this group is worth the same. These are the ones whose verdict changes what your business gets: the indexes people actually search and the assistants they actually ask.

FoundChatGPT-User is stopped by its own robots.txt rule (User-agent: chatgpt-user); Claude-SearchBot is stopped by its own robots.txt rule (User-agent: claude-searchbot); ClaudeBot is stopped by its own robots.txt rule (User-agent: claudebot); GPTBot is stopped by its own robots.txt rule (User-agent: gptbot); Google-Extended is stopped by its own robots.txt rule (User-agent: google-extended); OAI-SearchBot is stopped by its own robots.txt rule (User-agent: oai-searchbot); PerplexityBot is stopped by its own robots.txt rule (User-agent: perplexitybot); your server refused every test request, a plain browser's included, so its answer to DuckAssistBot is not counted here

Do thisFix these before anything else in this report: each one is a whole audience that cannot see the site today. The table below names the exact rule or answer responsible for each.

Worth knowing Judged on robots.txt alone

CheckedSome operators publish a robots.txt token but never a literal user-agent string. Those bots cannot be impersonated honestly, so they are judged on your rules only — no live request was made in their name.

Found9 of 33: ChatGPT Operator, ChatGPT-User/2.0, Google-Extended, Grok, GrokBot, MistralAI-User, MyCentralAIScraperBot, Perplexity-User and 1 more

Worth knowingNothing to do. Their robots.txt verdict is still exact — it is read from your own file — only the server-side half is unknown.

The AI assistants group, bot by bot

  1. ✗

    AI2Bot

    robots.txt: Allowed with exceptions · your server: HTTP 403

  2. ✗

    Ai2Bot-Dolma

    robots.txt: Allowed with exceptions · your server: HTTP 403

  3. ✓

    anthropic-ai

    robots.txt: Not allowed · your server: Redirected · its own robots.txt rule (User-agent: anthropic-ai)

  4. !

    ChatGPT Operator

    robots.txt: Not allowed · your server: Not probed · its own robots.txt rule (User-agent: chatgpt-user)

  5. ✓

    ChatGPT-User

    robots.txt: Not allowed · your server: Redirected · its own robots.txt rule (User-agent: chatgpt-user)

  6. !

    ChatGPT-User/2.0

    robots.txt: Not allowed · your server: Not probed · its own robots.txt rule (User-agent: chatgpt-user)

  7. ✓

    Claude-Code

    robots.txt: Allowed with exceptions · your server: Redirected

  8. ✓

    Claude-SearchBot

    robots.txt: Not allowed · your server: Redirected · its own robots.txt rule (User-agent: claude-searchbot)

  9. ✓

    Claude-User

    robots.txt: Not allowed · your server: Redirected · its own robots.txt rule (User-agent: claude-user)

  10. ✓

    Claude-Web

    robots.txt: Not allowed · your server: Redirected · its own robots.txt rule (User-agent: claude-web)

  11. ✓

    ClaudeBot

    robots.txt: Not allowed · your server: Redirected · its own robots.txt rule (User-agent: claudebot)

  12. ✓

    cohere-ai

    robots.txt: Not allowed · your server: Redirected · its own robots.txt rule (User-agent: cohere-ai)

  13. ✗

    cohere-training-data-crawler

    robots.txt: Allowed with exceptions · your server: HTTP 403

  14. ✗

    DeepSeekBot

    robots.txt: Allowed with exceptions · your server: HTTP 403

  15. ✗

    DigitalOceanGenAICrawler

    robots.txt: Allowed with exceptions · your server: HTTP 403

  16. ✗

    DuckAssistBot

    robots.txt: Allowed with exceptions · your server: HTTP 403

  17. !

    Google-Extended

    robots.txt: Not allowed · your server: Not probed · its own robots.txt rule (User-agent: google-extended)

  18. ✓

    GPTBot

    robots.txt: Not allowed · your server: Redirected · its own robots.txt rule (User-agent: gptbot)

  19. !

    Grok

    robots.txt: Allowed with exceptions · your server: Not probed

  20. !

    GrokBot

    robots.txt: Allowed with exceptions · your server: Not probed

  21. ✗

    img2dataset

    robots.txt: Allowed with exceptions · your server: HTTP 403

  22. ✗

    LINER Bot

    robots.txt: Allowed with exceptions · your server: HTTP 403

  23. !

    MistralAI-User

    robots.txt: Not allowed · your server: Not probed · its own robots.txt rule (User-agent: mistralai-user)

  24. ✓

    MistralAI-User/1.0

    robots.txt: Not allowed · your server: Redirected · its own robots.txt rule (User-agent: mistralai-user)

  25. !

    MyCentralAIScraperBot

    robots.txt: Allowed with exceptions · your server: Not probed

  26. ✓

    OAI-SearchBot

    robots.txt: Not allowed · your server: Redirected · its own robots.txt rule (User-agent: oai-searchbot)

  27. ✗

    PanguBot

    robots.txt: Allowed with exceptions · your server: HTTP 403

  28. !

    Perplexity-User

    robots.txt: Not allowed · your server: Not probed · its own robots.txt rule (User-agent: perplexity-user)

  29. ✓

    Perplexity-User/1.0

    robots.txt: Not allowed · your server: Redirected · its own robots.txt rule (User-agent: perplexity-user)

  30. ✓

    PerplexityBot

    robots.txt: Not allowed · your server: Redirected · its own robots.txt rule (User-agent: perplexitybot)

  31. !

    quillbot.com

    robots.txt: Allowed with exceptions · your server: Not probed

  32. ✗

    SBIntuitionsBot

    robots.txt: Allowed with exceptions · your server: HTTP 403

  33. ✓

    YouBot

    robots.txt: Not allowed · your server: Redirected · its own robots.txt rule (User-agent: youbot)

See the roster table
Every bot of the AI assistants group, with its two verdicts
Bot Gate 1 · your robots.txt Gate 2 · your server, live What stopped it
AI2Bot
A bare token.
Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +99
HTTP 403 the server (HTTP 403)
Ai2Bot-Dolma Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +99
HTTP 403 the server (HTTP 403)
anthropic-ai
alias — same bot as another row
A deprecated legacy token, superseded by ClaudeBot.
Not allowed
by User-agent: anthropic-ai
Redirected its own robots.txt rule (User-agent: anthropic-ai)
ChatGPT Operator
alias — same bot as another row
Folded into ChatGPT agent in 2025. It sends no distinctive user-agent -- it uses an ordinary Chrome desktop string and self-identifies with a Signature-Agent: https://chatgpt.com header, so it cannot be impersonated by user-agent alone.
Not allowed
by User-agent: chatgpt-user
Not probed its own robots.txt rule (User-agent: chatgpt-user)
ChatGPT-User
OpenAI's live fetch when a ChatGPT user asks about your page.
Not allowed
by User-agent: chatgpt-user
Redirected its own robots.txt rule (User-agent: chatgpt-user)
ChatGPT-User/2.0
alias — same bot as another row
OpenAI's published version is 1.0; there is no evidence of a 2.0, so only the token is checked.
Not allowed
by User-agent: chatgpt-user
Not probed its own robots.txt rule (User-agent: chatgpt-user)
Claude-Code
alias — same bot as another row
Not a separately documented crawler: the Claude Code CLI's own fetches ride the Claude-User pattern.
Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +99
Redirected —
Claude-SearchBot Not allowed
by User-agent: claude-searchbot
Redirected its own robots.txt rule (User-agent: claude-searchbot)
Claude-User Not allowed
by User-agent: claude-user
Redirected its own robots.txt rule (User-agent: claude-user)
Claude-Web
alias — same bot as another row
Deprecated, replaced by ClaudeBot / Claude-User / Claude-SearchBot. The string is historical.
Not allowed
by User-agent: claude-web
Redirected its own robots.txt rule (User-agent: claude-web)
ClaudeBot Not allowed
by User-agent: claudebot
Redirected its own robots.txt rule (User-agent: claudebot)
cohere-ai
A bare token; no Mozilla-wrapped form is documented.
Not allowed
by User-agent: cohere-ai
Redirected its own robots.txt rule (User-agent: cohere-ai)
cohere-training-data-crawler Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +99
HTTP 403 the server (HTTP 403)
DeepSeekBot Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +99
HTTP 403 the server (HTTP 403)
DigitalOceanGenAICrawler
Names conflict: DigitalOcean's current docs say DigitalOceanGradientAICrawler/1.0, a widely cited third-party report says DigitalOceanGenAICrawler/1.0.
Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +99
HTTP 403 the server (HTTP 403)
DuckAssistBot Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +99
HTTP 403 the server (HTTP 403)
Google-Extended
A robots.txt token only: the crawl happens under the standard Googlebot or GoogleOther user-agent. It governs whether your pages may ground Gemini answers and AI Overviews.
Not allowed
by User-agent: google-extended
Not probed its own robots.txt rule (User-agent: google-extended)
GPTBot Not allowed
by User-agent: gptbot
Redirected its own robots.txt rule (User-agent: gptbot)
Grok
xAI publishes no crawler documentation page; third-party trackers disagree (Grok, xAI-Grok/1.0, GrokBot/1.0), so no string is sent.
Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +99
Not probed —
GrokBot
alias — same bot as another row
Same caveat as Grok: no official documentation.
Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +99
Not probed —
img2dataset
An open-source tool run by many independent operators.
Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +99
HTTP 403 the server (HTTP 403)
LINER Bot Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +99
HTTP 403 the server (HTTP 403)
MistralAI-User
alias — same bot as another row
1.0 is the only published version.
Not allowed
by User-agent: mistralai-user
Not probed its own robots.txt rule (User-agent: mistralai-user)
MistralAI-User/1.0 Not allowed
by User-agent: mistralai-user
Redirected its own robots.txt rule (User-agent: mistralai-user)
MyCentralAIScraperBot
The token circulates in blocklists; the operator and the user-agent string are undocumented.
Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +99
Not probed —
OAI-SearchBot
Builds the index behind ChatGPT search results.
Not allowed
by User-agent: oai-searchbot
Redirected its own robots.txt rule (User-agent: oai-searchbot)
PanguBot
Huawei's Pangu LLM crawler.
Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +99
HTTP 403 the server (HTTP 403)
Perplexity-User
alias — same bot as another row
Same bot as Perplexity-User/1.0.
Not allowed
by User-agent: perplexity-user
Not probed its own robots.txt rule (User-agent: perplexity-user)
Perplexity-User/1.0 Not allowed
by User-agent: perplexity-user
Redirected its own robots.txt rule (User-agent: perplexity-user)
PerplexityBot Not allowed
by User-agent: perplexitybot
Redirected its own robots.txt rule (User-agent: perplexitybot)
quillbot.com
The robots.txt token is confirmed; no literal user-agent string is published.
Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +99
Not probed —
SBIntuitionsBot Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +99
HTTP 403 the server (HTTP 403)
YouBot Not allowed
by User-agent: youbot
Redirected its own robots.txt rule (User-agent: youbot)

Section D · Scored group

Google's own robots

Critical

0 of 10 bots read your site.

Measured Nov. 16, 2026, 2:03 a.m.

0/3 scored points pass. Weakest point: Server-crawl-live-test — the live HEAD request (0%).

Your robots.txt lets 3 of 10 in. Your server says no to 8.

10 bots knock Google's own robots

Gate 1 Your robots.txt

Allowed with exceptionsNot allowed

Your file names and blocks GoogleOther, GoogleOther-Image, GoogleOther-Video, Googlebot-Discovery, Googlebot-Image, Googlebot-News, Googlebot-Video. 3 of them are allowed with exceptions — the closed paths are housekeeping only, no known content page is among them.

Gate 2 Your server, live

Captcha wallRefusedNot probed

8 Turned away by the server Gemini-Deep-Research HTTP 403Google-InspectionTool HTTP 403Storebot-Google HTTP 403GoogleOther CaptchaGoogleOther-Image CaptchaGoogleOther-Video CaptchaGooglebot-Image CaptchaGooglebot-Video Captcha
2 Judged on the file alone Googlebot-DiscoveryGooglebot-News

Their operator publishes no user-agent string, so no honest knock is possible.

What the measurement means, point by point

Critical Server-robots-instructions — what robots.txt permits 30%

CheckedThe first thing every well-behaved bot reads. A Disallow rule on the homepage stops it before it ever requests a page, so nothing further down can compensate. A group that closes some paths but not the homepage is 'allowed with exceptions' — it counts as allowed, and the exceptions are tested against your known content pages below. A bot no rule names is allowed — silence is permission — and shown as 'not indicated'.

Found7 of 10 not allowed: GoogleOther, GoogleOther-Image, GoogleOther-Video, Googlebot-Discovery, Googlebot-Image, Googlebot-News, Googlebot-Video; 3 of them are allowed with exceptions — the closed paths are housekeeping only, no known content page is among them.

Do thisYour robots.txt closes the door to GoogleOther, GoogleOther-Image, GoogleOther-Video, Googlebot-Discovery, Googlebot-Image, Googlebot-News, Googlebot-Video. Open the file and look for the User-agent group named in the table below: unless the block is deliberate, delete its Disallow: / line. This is a one-line change with an outsized effect.

Critical Server-crawl-live-test — the live HEAD request 0%

CheckedWhat your server really answers to one HEAD request wearing that bot's name. A firewall, a CDN rule or a bot-protection product can refuse a crawler your robots.txt welcomes — and robots.txt alone can never reveal it.

Found8 forbidden: Gemini-Deep-Research, Google-InspectionTool, GoogleOther, GoogleOther-Image, GoogleOther-Video, Googlebot-Image, Googlebot-Video, Storebot-Google

Do thisRead this one before acting: an ordinary desktop browser was refused too, so your server is not singling out crawlers — it is refusing our test requests wholesale (a rate limit, a geo or IP rule, or a challenge page). Check what your firewall does with unfamiliar visitors before changing any bot rule.

Worth knowing Do the two tests agree?

CheckedWhether the server-robots-instructions and the server-crawl-live-test tell the same story. They often do not: robots.txt is a published promise, the live answer is what actually happens, and only the comparison shows which one to believe.

Found3 bot(s) invited by robots.txt but refused by the server: Gemini-Deep-Research, Google-InspectionTool, Storebot-Google

Worth knowingA contradiction is worth resolving whichever way you meant it. Invited-but-refused: your firewall overrules your robots.txt — align them, or accept that the file is advertising a welcome the server does not honour. Blocked-but-served: your robots.txt line is a polite request only; a bot that ignores it gets the page, so add a server rule if the block really matters.

Critical The bots that decide the outcome 33%

CheckedNot every bot in this group is worth the same. These are the ones whose verdict changes what your business gets: the indexes people actually search and the assistants they actually ask.

FoundGoogleOther is stopped by its own robots.txt rule (User-agent: googleother); Googlebot-Image is stopped by its own robots.txt rule (User-agent: googlebot); your server refused every test request, a plain browser's included, so its answer to Google-InspectionTool is not counted here

Do thisFix these before anything else in this report: each one is a whole audience that cannot see the site today. The table below names the exact rule or answer responsible for each.

Worth knowing Judged on robots.txt alone

CheckedSome operators publish a robots.txt token but never a literal user-agent string. Those bots cannot be impersonated honestly, so they are judged on your rules only — no live request was made in their name.

Found2 of 10: Googlebot-Discovery, Googlebot-News

Worth knowingNothing to do. Their robots.txt verdict is still exact — it is read from your own file — only the server-side half is unknown.

The Google's own robots group, bot by bot

  1. ✗

    Gemini-Deep-Research

    robots.txt: Allowed with exceptions · your server: HTTP 403

  2. ✗

    Google-InspectionTool

    robots.txt: Allowed with exceptions · your server: HTTP 403

  3. !

    Googlebot-Discovery

    robots.txt: Not allowed · your server: Not probed · its own robots.txt rule (User-agent: googlebot)

  4. ✗

    Googlebot-Image

    robots.txt: Not allowed · your server: Captcha · its own robots.txt rule (User-agent: googlebot)

  5. !

    Googlebot-News

    robots.txt: Not allowed · your server: Not probed · its own robots.txt rule (User-agent: googlebot)

  6. ✗

    Googlebot-Video

    robots.txt: Not allowed · your server: Captcha · its own robots.txt rule (User-agent: googlebot)

  7. ✗

    GoogleOther

    robots.txt: Not allowed · your server: Captcha · its own robots.txt rule (User-agent: googleother)

  8. ✗

    GoogleOther-Image

    robots.txt: Not allowed · your server: Captcha · its own robots.txt rule (User-agent: googleother)

  9. ✗

    GoogleOther-Video

    robots.txt: Not allowed · your server: Captcha · its own robots.txt rule (User-agent: googleother)

  10. ✗

    Storebot-Google

    robots.txt: Allowed with exceptions · your server: HTTP 403

See the roster table
Every bot of the Google's own robots group, with its two verdicts
Bot Gate 1 · your robots.txt Gate 2 · your server, live What stopped it
Gemini-Deep-Research Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +99
HTTP 403 the server (HTTP 403)
Google-InspectionTool
What Search Console's URL Inspection uses. A mobile variant wraps it in a Nexus 5X Chrome string.
Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +99
HTTP 403 the server (HTTP 403)
Googlebot-Discovery
alias — same bot as another row
No documentation found for this exact name. Discover surfacing uses the standard Googlebot / Googlebot-Image, so this is likely a naming confusion.
Not allowed
by User-agent: googlebot
Not probed its own robots.txt rule (User-agent: googlebot)
Googlebot-Image Not allowed
by User-agent: googlebot
Captcha its own robots.txt rule (User-agent: googlebot)
Googlebot-News
alias — same bot as another row
A distinct robots.txt token with no separate string: the requests use the standard Googlebot user-agent.
Not allowed
by User-agent: googlebot
Not probed its own robots.txt rule (User-agent: googlebot)
Googlebot-Video Not allowed
by User-agent: googlebot
Captcha its own robots.txt rule (User-agent: googlebot)
GoogleOther
A bare token; may also appear via standard Googlebot-style strings.
Not allowed
by User-agent: googleother
Captcha its own robots.txt rule (User-agent: googleother)
GoogleOther-Image Not allowed
by User-agent: googleother
Captcha its own robots.txt rule (User-agent: googleother)
GoogleOther-Video Not allowed
by User-agent: googleother
Captcha its own robots.txt rule (User-agent: googleother)
Storebot-Google
An Android/mobile variant also exists.
Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +99
HTTP 403 the server (HTTP 403)

Section E

Other robots

Fix soon

Social Bots

Fix soon

7 of 11 bots read your site.

Measured Nov. 16, 2026, 2:03 a.m.

1/3 scored points pass. Weakest point: Server-crawl-live-test — the live HEAD request (64%).

Your robots.txt lets 8 of 11 in. Your server says no to 4.

11 bots knock Social Bots

Gate 1 Your robots.txt

Allowed with exceptionsNot allowed

Your file names and blocks FacebookBot, LinkedInBot, Meta-ExternalAgent. 8 of them are allowed with exceptions — the closed paths are housekeeping only, no known content page is among them.

Gate 2 Your server, live

ServedRefused

7 Read your site FacebookBotLinkedInBotMeta-ExternalAgentPinterestbotSlackbotTwitterbotfacebookexternalhit
4 Turned away by the server Meta-ExternalAds HTTP 403Meta-ExternalFetcher HTTP 403Meta-WebIndexer HTTP 403Quora-Bot HTTP 403

What the measurement means, point by point

Fix soon Server-robots-instructions — what robots.txt permits 73%

CheckedThe first thing every well-behaved bot reads. A Disallow rule on the homepage stops it before it ever requests a page, so nothing further down can compensate. A group that closes some paths but not the homepage is 'allowed with exceptions' — it counts as allowed, and the exceptions are tested against your known content pages below. A bot no rule names is allowed — silence is permission — and shown as 'not indicated'.

Found3 of 11 not allowed: FacebookBot, LinkedInBot, Meta-ExternalAgent; 8 of them are allowed with exceptions — the closed paths are housekeeping only, no known content page is among them.

Do thisYour robots.txt closes the door to FacebookBot, LinkedInBot, Meta-ExternalAgent. Open the file and look for the User-agent group named in the table below: unless the block is deliberate, delete its Disallow: / line. This is a one-line change with an outsized effect.

Fix soon Server-crawl-live-test — the live HEAD request 64%

CheckedWhat your server really answers to one HEAD request wearing that bot's name. A firewall, a CDN rule or a bot-protection product can refuse a crawler your robots.txt welcomes — and robots.txt alone can never reveal it.

Found4 forbidden: Meta-ExternalAds, Meta-ExternalFetcher, Meta-WebIndexer, Quora-Bot

Do thisRead this one before acting: an ordinary desktop browser was refused too, so your server is not singling out crawlers — it is refusing our test requests wholesale (a rate limit, a geo or IP rule, or a challenge page). Check what your firewall does with unfamiliar visitors before changing any bot rule.

Worth knowing Indexing directives in the headers

CheckedThe X-Robots-Tag response header of the answers your server actually served. A noindex here lets the bot in and then tells it to throw the page away — the most expensive way to be invisible, because it looks fine in every other check. Only served answers are read: a wall's or an error page's header is not yours.

Foundno X-Robots-Tag header in any served answer

Worth knowing Do the two tests agree?

CheckedWhether the server-robots-instructions and the server-crawl-live-test tell the same story. They often do not: robots.txt is a published promise, the live answer is what actually happens, and only the comparison shows which one to believe.

Found4 bot(s) invited by robots.txt but refused by the server — the same ones the live test above names; 3 bot(s) blocked in robots.txt but served anyway: FacebookBot, LinkedInBot, Meta-ExternalAgent

Worth knowingA contradiction is worth resolving whichever way you meant it. Invited-but-refused: your firewall overrules your robots.txt — align them, or accept that the file is advertising a welcome the server does not honour. Blocked-but-served: your robots.txt line is a polite request only; a bot that ignores it gets the page, so add a server rule if the block really matters.

Good The bots that decide the outcome 80%

CheckedNot every bot in this group is worth the same. These are the ones whose verdict changes what your business gets: the indexes people actually search and the assistants they actually ask.

FoundLinkedInBot is stopped by its own robots.txt rule (User-agent: linkedinbot)

The Social Bots group, bot by bot

  1. ✓

    FacebookBot

    robots.txt: Not allowed · your server: Redirected · its own robots.txt rule (User-agent: facebookbot)

  2. ✓

    facebookexternalhit

    robots.txt: Allowed with exceptions · your server: Redirected

  3. ✓

    LinkedInBot

    robots.txt: Not allowed · your server: Redirected · its own robots.txt rule (User-agent: linkedinbot)

  4. ✗

    Meta-ExternalAds

    robots.txt: Allowed with exceptions · your server: HTTP 403

  5. ✓

    Meta-ExternalAgent

    robots.txt: Not allowed · your server: Redirected · its own robots.txt rule (User-agent: meta-externalagent)

  6. ✗

    Meta-ExternalFetcher

    robots.txt: Allowed with exceptions · your server: HTTP 403

  7. ✗

    Meta-WebIndexer

    robots.txt: Allowed with exceptions · your server: HTTP 403

  8. ✓

    Pinterestbot

    robots.txt: Allowed with exceptions · your server: Redirected

  9. ✗

    Quora-Bot

    robots.txt: Allowed with exceptions · your server: HTTP 403

  10. ✓

    Slackbot

    robots.txt: Allowed with exceptions · your server: Redirected

  11. ✓

    Twitterbot

    robots.txt: Allowed with exceptions · your server: Redirected

See the roster table
Every bot of the Social Bots group, with its two verdicts
Bot Gate 1 · your robots.txt Gate 2 · your server, live What stopped it
FacebookBot
Meta's general research/product crawler, distinct from facebookexternalhit.
Not allowed
by User-agent: facebookbot
Redirected its own robots.txt rule (User-agent: facebookbot)
facebookexternalhit
The link-preview fetcher; also seen bare as facebookexternalhit/1.1.
Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +99
Redirected —
LinkedInBot
The HttpClient version field varies.
Not allowed
by User-agent: linkedinbot
Redirected its own robots.txt rule (User-agent: linkedinbot)
Meta-ExternalAds
Ad-landing-page review.
Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +99
HTTP 403 the server (HTTP 403)
Meta-ExternalAgent
AI training plus search indexing.
Not allowed
by User-agent: meta-externalagent
Redirected its own robots.txt rule (User-agent: meta-externalagent)
Meta-ExternalFetcher
On-demand fetch for Meta AI features.
Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +99
HTTP 403 the server (HTTP 403)
Meta-WebIndexer
Meta's newer (2025) general web-indexing crawler.
Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +99
HTTP 403 the server (HTTP 403)
Pinterestbot
A lighter fetcher is also seen: Pinterest/0.2 (+https://www.pinterest.com/bot.html).
Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +99
Redirected —
Quora-Bot Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +99
HTTP 403 the server (HTTP 403)
Slackbot
Other Slack integrations use Slackbot 1.0 (+https://api.slack.com/robots).
Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +99
Redirected —
Twitterbot Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +99
Redirected —

SEO Tools

0 of 21 bots read your site.

Measured Nov. 16, 2026, 2:03 a.m.

Your robots.txt says yes to all 21. Your server says no to 20.

21 bots knock SEO Tools

Gate 1 Your robots.txt

All of them ride on your catch-all group (User-agent: *) — no rule in your file names one of them individually. 21 of them are allowed with exceptions — the closed paths are housekeeping only, no known content page is among them.

Gate 2 Your server, live

RefusedNot probed

20 Turned away by the server AhrefsBot HTTP 403AudigentAdBot HTTP 403AwarioRssBot HTTP 403AwarioSmartBot HTTP 403Brandwatch HTTP 403Chrome-Lighthouse HTTP 403DataForSeoBot HTTP 403DotBot HTTP 403Google Page Speed Insights HTTP 403Jetslide HTTP 403 + 10 more
1 Judged on the file alone NeticleBot

Their operator publishes no user-agent string, so no honest knock is possible.

What the measurement means, point by point

Worth knowing Server-robots-instructions — what robots.txt permits

CheckedThe first thing every well-behaved bot reads. A Disallow rule on the homepage stops it before it ever requests a page, so nothing further down can compensate. A group that closes some paths but not the homepage is 'allowed with exceptions' — it counts as allowed, and the exceptions are tested against your known content pages below. A bot no rule names is allowed — silence is permission — and shown as 'not indicated'.

Foundall 21 are allowed by robots.txt; 21 of them are allowed with exceptions — the closed paths are housekeeping only, no known content page is among them.

Worth knowing Server-crawl-live-test — the live HEAD request

CheckedWhat your server really answers to one HEAD request wearing that bot's name. A firewall, a CDN rule or a bot-protection product can refuse a crawler your robots.txt welcomes — and robots.txt alone can never reveal it.

Found20 forbidden: AhrefsBot, AudigentAdBot, AwarioRssBot, AwarioSmartBot, Brandwatch, Chrome-Lighthouse, DataForSeoBot, DotBot and 12 more

Worth knowingRead this one before acting: an ordinary desktop browser was refused too, so your server is not singling out crawlers — it is refusing our test requests wholesale (a rate limit, a geo or IP rule, or a challenge page). Check what your firewall does with unfamiliar visitors before changing any bot rule.

Worth knowing Do the two tests agree?

CheckedWhether the server-robots-instructions and the server-crawl-live-test tell the same story. They often do not: robots.txt is a published promise, the live answer is what actually happens, and only the comparison shows which one to believe.

Found20 bot(s) invited by robots.txt but refused by the server — the same ones the live test above names

Worth knowingA contradiction is worth resolving whichever way you meant it. Invited-but-refused: your firewall overrules your robots.txt — align them, or accept that the file is advertising a welcome the server does not honour. Blocked-but-served: your robots.txt line is a polite request only; a bot that ignores it gets the page, so add a server rule if the block really matters.

Worth knowing Judged on robots.txt alone

CheckedSome operators publish a robots.txt token but never a literal user-agent string. Those bots cannot be impersonated honestly, so they are judged on your rules only — no live request was made in their name.

Found1 of 21: NeticleBot

Worth knowingNothing to do. Their robots.txt verdict is still exact — it is read from your own file — only the server-side half is unknown.

The SEO Tools group, bot by bot

  1. ✗

    AhrefsBot

    robots.txt: Allowed with exceptions · your server: HTTP 403

  2. ✗

    AudigentAdBot

    robots.txt: Allowed with exceptions · your server: HTTP 403

  3. ✗

    AwarioRssBot

    robots.txt: Allowed with exceptions · your server: HTTP 403

  4. ✗

    AwarioSmartBot

    robots.txt: Allowed with exceptions · your server: HTTP 403

  5. ✗

    Brandwatch

    robots.txt: Allowed with exceptions · your server: HTTP 403

  6. ✗

    Chrome-Lighthouse

    robots.txt: Allowed with exceptions · your server: HTTP 403

  7. ✗

    DataForSeoBot

    robots.txt: Allowed with exceptions · your server: HTTP 403

  8. ✗

    DotBot

    robots.txt: Allowed with exceptions · your server: HTTP 403

  9. ✗

    Google Page Speed Insights

    robots.txt: Allowed with exceptions · your server: HTTP 403

  10. ✗

    Jetslide

    robots.txt: Allowed with exceptions · your server: HTTP 403

  11. ✗

    Meltwater

    robots.txt: Allowed with exceptions · your server: HTTP 403

  12. ✗

    MJ12bot

    robots.txt: Allowed with exceptions · your server: HTTP 403

  13. !

    NeticleBot

    robots.txt: Allowed with exceptions · your server: Not probed

  14. ✗

    Netvibes

    robots.txt: Allowed with exceptions · your server: HTTP 403

  15. ✗

    peer39_crawler

    robots.txt: Allowed with exceptions · your server: HTTP 403

  16. ✗

    Screaming Frog SEO Spider

    robots.txt: Allowed with exceptions · your server: HTTP 403

  17. ✗

    SearchmetricsBot

    robots.txt: Allowed with exceptions · your server: HTTP 403

  18. ✗

    SemrushBot

    robots.txt: Allowed with exceptions · your server: HTTP 403

  19. ✗

    SemrushBot-OCOB

    robots.txt: Allowed with exceptions · your server: HTTP 403

  20. ✗

    SemrushBotSwa

    robots.txt: Allowed with exceptions · your server: HTTP 403

  21. ✗

    Sidetrade Indexer Bot

    robots.txt: Allowed with exceptions · your server: HTTP 403

See the roster table
Every bot of the SEO Tools group, with its two verdicts
Bot Gate 1 · your robots.txt Gate 2 · your server, live What stopped it
AhrefsBot Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +99
HTTP 403 the server (HTTP 403)
AudigentAdBot Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +99
HTTP 403 the server (HTTP 403)
AwarioRssBot Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +99
HTTP 403 the server (HTTP 403)
AwarioSmartBot Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +99
HTTP 403 the server (HTTP 403)
Brandwatch
The vendor's own page says the string varies, so this literal is low confidence.
Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +99
HTTP 403 the server (HTTP 403)
Chrome-Lighthouse Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +99
HTTP 403 the server (HTTP 403)
DataForSeoBot
The crawler behind the data AkOOre's own reports are built on.
Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +99
HTTP 403 the server (HTTP 403)
DotBot
A newer DotBot/2.0; +https://dotbot.com/about/ form is also reported but unconfirmed.
Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +99
HTTP 403 the server (HTTP 403)
Google Page Speed Insights
alias — same bot as another row
The documented legacy form. Recent PSI requests reportedly drop these tokens for a plain Chrome user-agent.
Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +99
HTTP 403 the server (HTTP 403)
Jetslide Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +99
HTTP 403 the server (HTTP 403)
Meltwater Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +99
HTTP 403 the server (HTTP 403)
MJ12bot
Newer v2.0.x builds are also reported, same format.
Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +99
HTTP 403 the server (HTTP 403)
NeticleBot
The vendor has a bot-info page but quotes no exact header.
Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +99
Not probed —
Netvibes Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +99
HTTP 403 the server (HTTP 403)
peer39_crawler
Ad-context classification. The source list names it twice (SEO Tools and Other Agents); it is one bot, listed once.
Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +99
HTTP 403 the server (HTTP 403)
Screaming Frog SEO Spider
The version tracks the installed app and changes every release.
Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +99
HTTP 403 the server (HTTP 403)
SearchmetricsBot Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +99
HTTP 403 the server (HTTP 403)
SemrushBot
Older SemrushBot-BA / -SA / -SI variants also exist.
Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +99
HTTP 403 the server (HTTP 403)
SemrushBot-OCOB
The Content Toolkit crawler.
Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +99
HTTP 403 the server (HTTP 403)
SemrushBotSwa Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +99
HTTP 403 the server (HTTP 403)
Sidetrade Indexer Bot Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +99
HTTP 403 the server (HTTP 403)

Scrapers

1 of 46 bots read your site.

Measured Nov. 16, 2026, 2:03 a.m.

Your robots.txt says yes to all 46. Your server says no to 34.

46 bots knock Scrapers

Gate 1 Your robots.txt

All of them ride on your catch-all group (User-agent: *) — no rule in your file names one of them individually. 46 of them are allowed with exceptions — the closed paths are housekeeping only, no known content page is among them.

Gate 2 Your server, live

ServedCaptcha wallRefusedNot probed

1 Read your site Barkrowler
34 Turned away by the server Arquivo-web-crawler HTTP 403BLEXBot HTTP 403Bytespider HTTP 403Cotoyogi HTTP 403Crawl4AI HTTP 403Diffbot HTTP 403EchoboxBot HTTP 403Factset_spyderbot HTTP 403Firecrawl HTTP 403FriendlyCrawler HTTP 403 + 24 more
11 Judged on the file alone Amazon KendraBravestCrawlspaceEchobot BotJenkersBotPoseidon Research CrawlerSeekrSeekrBothada.newsimediaethics.org + 1 more

Their operator publishes no user-agent string, so no honest knock is possible.

What the measurement means, point by point

Worth knowing Server-robots-instructions — what robots.txt permits

CheckedThe first thing every well-behaved bot reads. A Disallow rule on the homepage stops it before it ever requests a page, so nothing further down can compensate. A group that closes some paths but not the homepage is 'allowed with exceptions' — it counts as allowed, and the exceptions are tested against your known content pages below. A bot no rule names is allowed — silence is permission — and shown as 'not indicated'.

Foundall 46 are allowed by robots.txt; 46 of them are allowed with exceptions — the closed paths are housekeeping only, no known content page is among them.

Worth knowing Server-crawl-live-test — the live HEAD request

CheckedWhat your server really answers to one HEAD request wearing that bot's name. A firewall, a CDN rule or a bot-protection product can refuse a crawler your robots.txt welcomes — and robots.txt alone can never reveal it.

Found34 forbidden: Arquivo-web-crawler, BLEXBot, Bytespider, CCBot, Cotoyogi, Crawl4AI, Diffbot, EchoboxBot and 26 more

Worth knowingRead this one before acting: an ordinary desktop browser was refused too, so your server is not singling out crawlers — it is refusing our test requests wholesale (a rate limit, a geo or IP rule, or a challenge page). Check what your firewall does with unfamiliar visitors before changing any bot rule.

Worth knowing Indexing directives in the headers

CheckedThe X-Robots-Tag response header of the answers your server actually served. A noindex here lets the bot in and then tells it to throw the page away — the most expensive way to be invisible, because it looks fine in every other check. Only served answers are read: a wall's or an error page's header is not yours.

Foundno X-Robots-Tag header in any served answer

Worth knowing Do the two tests agree?

CheckedWhether the server-robots-instructions and the server-crawl-live-test tell the same story. They often do not: robots.txt is a published promise, the live answer is what actually happens, and only the comparison shows which one to believe.

Found34 bot(s) invited by robots.txt but refused by the server — the same ones the live test above names

Worth knowingA contradiction is worth resolving whichever way you meant it. Invited-but-refused: your firewall overrules your robots.txt — align them, or accept that the file is advertising a welcome the server does not honour. Blocked-but-served: your robots.txt line is a polite request only; a bot that ignores it gets the page, so add a server rule if the block really matters.

Worth knowing Judged on robots.txt alone

CheckedSome operators publish a robots.txt token but never a literal user-agent string. Those bots cannot be impersonated honestly, so they are judged on your rules only — no live request was made in their name.

Found11 of 46: Amazon Kendra, Bravest, Crawlspace, Echobot Bot, JenkersBot, Poseidon Research Crawler, Seekr, SeekrBot and 3 more

Worth knowingNothing to do. Their robots.txt verdict is still exact — it is read from your own file — only the server-side half is unknown.

The Scrapers group, bot by bot

  1. !

    Amazon Kendra

    robots.txt: Allowed with exceptions · your server: Not probed

  2. ✗

    Arquivo-web-crawler

    robots.txt: Allowed with exceptions · your server: HTTP 403

  3. ✓

    Barkrowler

    robots.txt: Allowed with exceptions · your server: Redirected

  4. ✗

    BLEXBot

    robots.txt: Allowed with exceptions · your server: HTTP 403

  5. !

    Bravest

    robots.txt: Allowed with exceptions · your server: Not probed

  6. ✗

    Bytespider

    robots.txt: Allowed with exceptions · your server: HTTP 403

  7. ✗

    CCBot

    robots.txt: Allowed with exceptions · your server: Captcha · a bot wall instead of the page — redirected to a “captcha” page

  8. ✗

    coccocbot-web

    robots.txt: Allowed with exceptions · your server: HTTP 403

  9. ✗

    Cotoyogi

    robots.txt: Allowed with exceptions · your server: HTTP 403

  10. ✗

    Crawl4AI

    robots.txt: Allowed with exceptions · your server: HTTP 403

  11. ✗

    crawler4j

    robots.txt: Allowed with exceptions · your server: HTTP 403

  12. !

    Crawlspace

    robots.txt: Allowed with exceptions · your server: Not probed

  13. ✗

    Diffbot

    robots.txt: Allowed with exceptions · your server: HTTP 403

  14. !

    Echobot Bot

    robots.txt: Allowed with exceptions · your server: Not probed

  15. ✗

    EchoboxBot

    robots.txt: Allowed with exceptions · your server: HTTP 403

  16. ✗

    Factset_spyderbot

    robots.txt: Allowed with exceptions · your server: HTTP 403

  17. ✗

    Firecrawl

    robots.txt: Allowed with exceptions · your server: HTTP 403

  18. ✗

    FriendlyCrawler

    robots.txt: Allowed with exceptions · your server: HTTP 403

  19. !

    hada.news

    robots.txt: Allowed with exceptions · your server: Not probed

  20. ✗

    iaskspider

    robots.txt: Allowed with exceptions · your server: HTTP 403

  21. ✗

    ICC-Crawler

    robots.txt: Allowed with exceptions · your server: HTTP 403

  22. ✗

    ImagesiftBot

    robots.txt: Allowed with exceptions · your server: HTTP 403

  23. !

    imediaethics.org

    robots.txt: Allowed with exceptions · your server: Not probed

  24. ✗

    imgproxy

    robots.txt: Allowed with exceptions · your server: HTTP 403

  25. ✗

    ISSCyberRiskCrawler

    robots.txt: Allowed with exceptions · your server: HTTP 403

  26. !

    JenkersBot

    robots.txt: Allowed with exceptions · your server: Not probed

  27. ✗

    Kangaroo Bot

    robots.txt: Allowed with exceptions · your server: HTTP 403

  28. ✗

    magpie-crawler

    robots.txt: Allowed with exceptions · your server: HTTP 403

  29. !

    netEstate Imprint Crawler

    robots.txt: Allowed with exceptions · your server: Not probed

  30. ✗

    news-please

    robots.txt: Allowed with exceptions · your server: HTTP 403

  31. ✗

    NewsNow

    robots.txt: Allowed with exceptions · your server: HTTP 403

  32. ✗

    NovaAct

    robots.txt: Allowed with exceptions · your server: HTTP 403

  33. ✗

    omgili

    robots.txt: Allowed with exceptions · your server: HTTP 403

  34. ✗

    omgilibot

    robots.txt: Allowed with exceptions · your server: HTTP 403

  35. !

    Poseidon Research Crawler

    robots.txt: Allowed with exceptions · your server: Not probed

  36. ✗

    QualifiedBot

    robots.txt: Allowed with exceptions · your server: HTTP 403

  37. ✗

    Scrapy

    robots.txt: Allowed with exceptions · your server: HTTP 403

  38. ✗

    SeekportBot

    robots.txt: Allowed with exceptions · your server: HTTP 403

  39. !

    Seekr

    robots.txt: Allowed with exceptions · your server: Not probed

  40. !

    SeekrBot

    robots.txt: Allowed with exceptions · your server: Not probed

  41. ✗

    TaraGroup Intelligent Bot

    robots.txt: Allowed with exceptions · your server: HTTP 403

  42. ✗

    Timpibot

    robots.txt: Allowed with exceptions · your server: HTTP 403

  43. ✗

    Turnitin

    robots.txt: Allowed with exceptions · your server: HTTP 403

  44. ✗

    VelenPublicWebCrawler

    robots.txt: Allowed with exceptions · your server: HTTP 403

  45. ✗

    Webzio-Extended

    robots.txt: Allowed with exceptions · your server: HTTP 403

  46. ✗

    yacybot

    robots.txt: Allowed with exceptions · your server: HTTP 403

See the roster table
Every bot of the Scrapers group, with its two verdicts
Bot Gate 1 · your robots.txt Gate 2 · your server, live What stopped it
Amazon Kendra
AWS publishes the robots.txt token but no literal HTTP header.
Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +99
Not probed —
Arquivo-web-crawler Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +99
HTTP 403 the server (HTTP 403)
Barkrowler Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +99
Redirected —
BLEXBot Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +99
HTTP 403 the server (HTTP 403)
Bravest
No published user-agent string.
Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +99
Not probed —
Bytespider
ByteDance never published an official documentation page; the Bytespider token is the stable part.
Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +99
HTTP 403 the server (HTTP 403)
CCBot
Common Crawl feeds most open models' training sets.
Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +99
Captcha a bot wall instead of the page — redirected to a “captcha” page
coccocbot-web Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +99
HTTP 403 the server (HTTP 403)
Cotoyogi Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +99
HTTP 403 the server (HTTP 403)
Crawl4AI
A library default the caller can override.
Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +99
HTTP 403 the server (HTTP 403)
crawler4j
A library default, set per deployment by the developer.
Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +99
HTTP 403 the server (HTTP 403)
Crawlspace
No published user-agent string.
Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +99
Not probed —
Diffbot
Version numbers beyond 0.1 are unverified.
Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +99
HTTP 403 the server (HTTP 403)
Echobot Bot
Not to be confused with EchoboxBot above: different company, different bot.
Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +99
Not probed —
EchoboxBot
The hash segment is per-deployment.
Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +99
HTTP 403 the server (HTTP 403)
Factset_spyderbot Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +99
HTTP 403 the server (HTTP 403)
Firecrawl Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +99
HTTP 403 the server (HTTP 403)
FriendlyCrawler Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +99
HTTP 403 the server (HTTP 403)
hada.news
No published user-agent string.
Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +99
Not probed —
iaskspider Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +99
HTTP 403 the server (HTTP 403)
ICC-Crawler Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +99
HTTP 403 the server (HTTP 403)
ImagesiftBot Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +99
HTTP 403 the server (HTTP 403)
imediaethics.org
No published user-agent string.
Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +99
Not probed —
imgproxy
A self-hosted image-proxy tool rather than a crawler; the version tracks its release.
Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +99
HTTP 403 the server (HTTP 403)
ISSCyberRiskCrawler Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +99
HTTP 403 the server (HTTP 403)
JenkersBot
No published user-agent string.
Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +99
Not probed —
Kangaroo Bot
A differently spelled KangarooBot/1.0; +https://kangaroobot.com (no space) is documented elsewhere -- check which one matches your logs.
Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +99
HTTP 403 the server (HTTP 403)
magpie-crawler
A legacy crawler name, distinct from the newer Brandwatch user-agent in SEO Tools.
Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +99
HTTP 403 the server (HTTP 403)
netEstate Imprint Crawler
Not to be confused with the separately documented netEstate NE Crawler (+http://www.website-datenbank.de/), an older bot.
Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +99
Not probed —
news-please
A library default template, not a fixed signature.
Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +99
HTTP 403 the server (HTTP 403)
NewsNow Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +99
HTTP 403 the server (HTTP 403)
NovaAct Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +99
HTTP 403 the server (HTTP 403)
omgili Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +99
HTTP 403 the server (HTTP 403)
omgilibot
An older omgilibot/0.3 +http://www.omgili.com/Crawler.html form is also documented.
Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +99
HTTP 403 the server (HTTP 403)
Poseidon Research Crawler
Only the bare identifier is documented.
Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +99
Not probed —
QualifiedBot Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +99
HTTP 403 the server (HTTP 403)
Scrapy
The library default template; the version tracks the installed Scrapy release.
Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +99
HTTP 403 the server (HTTP 403)
SeekportBot Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +99
HTTP 403 the server (HTTP 403)
Seekr
alias — same bot as another row
No published user-agent string.
Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +99
Not probed —
SeekrBot
No published user-agent string.
Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +99
Not probed —
TaraGroup Intelligent Bot Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +99
HTTP 403 the server (HTTP 403)
Timpibot Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +99
HTTP 403 the server (HTTP 403)
Turnitin Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +99
HTTP 403 the server (HTTP 403)
VelenPublicWebCrawler Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +99
HTTP 403 the server (HTTP 403)
Webzio-Extended Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +99
HTTP 403 the server (HTTP 403)
yacybot
The bracketed system info varies per peer; yacybot is the stable token.
Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +99
HTTP 403 the server (HTTP 403)

Cloud Services

0 of 4 bots read your site.

Measured Nov. 16, 2026, 2:03 a.m.

Your robots.txt says yes to all 4. Your server says no to 4.

4 bots knock Cloud Services

Gate 1 Your robots.txt

All of them ride on your catch-all group (User-agent: *) — no rule in your file names one of them individually. 4 of them are allowed with exceptions — the closed paths are housekeeping only, no known content page is among them.

Gate 2 Your server, live

4 Turned away by the server AliyunSecBot HTTP 403Amazonbot HTTP 403Amzn-User HTTP 403Google-CloudVertexBot HTTP 403

What the measurement means, point by point

Worth knowing Server-robots-instructions — what robots.txt permits

CheckedThe first thing every well-behaved bot reads. A Disallow rule on the homepage stops it before it ever requests a page, so nothing further down can compensate. A group that closes some paths but not the homepage is 'allowed with exceptions' — it counts as allowed, and the exceptions are tested against your known content pages below. A bot no rule names is allowed — silence is permission — and shown as 'not indicated'.

Foundall 4 are allowed by robots.txt; 4 of them are allowed with exceptions — the closed paths are housekeeping only, no known content page is among them.

Worth knowing Server-crawl-live-test — the live HEAD request

CheckedWhat your server really answers to one HEAD request wearing that bot's name. A firewall, a CDN rule or a bot-protection product can refuse a crawler your robots.txt welcomes — and robots.txt alone can never reveal it.

Found4 forbidden: AliyunSecBot, Amazonbot, Amzn-User, Google-CloudVertexBot

Worth knowingRead this one before acting: an ordinary desktop browser was refused too, so your server is not singling out crawlers — it is refusing our test requests wholesale (a rate limit, a geo or IP rule, or a challenge page). Check what your firewall does with unfamiliar visitors before changing any bot rule.

Worth knowing Do the two tests agree?

CheckedWhether the server-robots-instructions and the server-crawl-live-test tell the same story. They often do not: robots.txt is a published promise, the live answer is what actually happens, and only the comparison shows which one to believe.

Found4 bot(s) invited by robots.txt but refused by the server — the same ones the live test above names

Worth knowingA contradiction is worth resolving whichever way you meant it. Invited-but-refused: your firewall overrules your robots.txt — align them, or accept that the file is advertising a welcome the server does not honour. Blocked-but-served: your robots.txt line is a polite request only; a bot that ignores it gets the page, so add a server rule if the block really matters.

The Cloud Services group, bot by bot

  1. ✗

    AliyunSecBot

    robots.txt: Allowed with exceptions · your server: HTTP 403

  2. ✗

    Amazonbot

    robots.txt: Allowed with exceptions · your server: HTTP 403

  3. ✗

    Amzn-User

    robots.txt: Allowed with exceptions · your server: HTTP 403

  4. ✗

    Google-CloudVertexBot

    robots.txt: Allowed with exceptions · your server: HTTP 403

See the roster table
Every bot of the Cloud Services group, with its two verdicts
Bot Gate 1 · your robots.txt Gate 2 · your server, live What stopped it
AliyunSecBot
An unusual format, but the canonical documented instance.
Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +99
HTTP 403 the server (HTTP 403)
Amazonbot
Real instances wrap this inside a full desktop or mobile Chrome-style user-agent.
Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +99
HTTP 403 the server (HTTP 403)
Amzn-User Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +99
HTTP 403 the server (HTTP 403)
Google-CloudVertexBot Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +99
HTTP 403 the server (HTTP 403)

Other Agents

3 of 5 bots read your site.

Measured Nov. 16, 2026, 2:03 a.m.

Your robots.txt says yes to all 5. Your server says no to 1.

5 bots knock Other Agents

Gate 1 Your robots.txt

All of them ride on your catch-all group (User-agent: *) — no rule in your file names one of them individually. 5 of them are allowed with exceptions — the closed paths are housekeeping only, no known content page is among them.

Gate 2 Your server, live

ServedRefusedNot probed

3 Read your site archive.org_botia_archiveria_archiver-web.archive.org
1 Turned away by the server TurnitinBot HTTP 403
1 Judged on the file alone ViennaTinyBot

Their operator publishes no user-agent string, so no honest knock is possible.

What the measurement means, point by point

Worth knowing Server-robots-instructions — what robots.txt permits

CheckedThe first thing every well-behaved bot reads. A Disallow rule on the homepage stops it before it ever requests a page, so nothing further down can compensate. A group that closes some paths but not the homepage is 'allowed with exceptions' — it counts as allowed, and the exceptions are tested against your known content pages below. A bot no rule names is allowed — silence is permission — and shown as 'not indicated'.

Foundall 5 are allowed by robots.txt; 5 of them are allowed with exceptions — the closed paths are housekeeping only, no known content page is among them.

Worth knowing Server-crawl-live-test — the live HEAD request

CheckedWhat your server really answers to one HEAD request wearing that bot's name. A firewall, a CDN rule or a bot-protection product can refuse a crawler your robots.txt welcomes — and robots.txt alone can never reveal it.

Found1 forbidden: TurnitinBot

Worth knowingRead this one before acting: an ordinary desktop browser was refused too, so your server is not singling out crawlers — it is refusing our test requests wholesale (a rate limit, a geo or IP rule, or a challenge page). Check what your firewall does with unfamiliar visitors before changing any bot rule.

Worth knowing Indexing directives in the headers

CheckedThe X-Robots-Tag response header of the answers your server actually served. A noindex here lets the bot in and then tells it to throw the page away — the most expensive way to be invisible, because it looks fine in every other check. Only served answers are read: a wall's or an error page's header is not yours.

Foundno X-Robots-Tag header in any served answer

Worth knowing Do the two tests agree?

CheckedWhether the server-robots-instructions and the server-crawl-live-test tell the same story. They often do not: robots.txt is a published promise, the live answer is what actually happens, and only the comparison shows which one to believe.

Found1 bot(s) invited by robots.txt but refused by the server — the same ones the live test above names

Worth knowingA contradiction is worth resolving whichever way you meant it. Invited-but-refused: your firewall overrules your robots.txt — align them, or accept that the file is advertising a welcome the server does not honour. Blocked-but-served: your robots.txt line is a polite request only; a bot that ignores it gets the page, so add a server rule if the block really matters.

Worth knowing Judged on robots.txt alone

CheckedSome operators publish a robots.txt token but never a literal user-agent string. Those bots cannot be impersonated honestly, so they are judged on your rules only — no live request was made in their name.

Found1 of 5: ViennaTinyBot

Worth knowingNothing to do. Their robots.txt verdict is still exact — it is read from your own file — only the server-side half is unknown.

The Other Agents group, bot by bot

  1. ✓

    archive.org_bot

    robots.txt: Allowed with exceptions · your server: Redirected

  2. ✓

    ia_archiver

    robots.txt: Allowed with exceptions · your server: Redirected

  3. ✓

    ia_archiver-web.archive.org

    robots.txt: Allowed with exceptions · your server: Redirected

  4. ✗

    TurnitinBot

    robots.txt: Allowed with exceptions · your server: HTTP 403

  5. !

    ViennaTinyBot

    robots.txt: Allowed with exceptions · your server: Not probed

See the roster table
Every bot of the Other Agents group, with its two verdicts
Bot Gate 1 · your robots.txt Gate 2 · your server, live What stopped it
archive.org_bot
Heritrix-version-qualified variants are also documented.
Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +99
Redirected —
ia_archiver
The historical Alexa string. Alexa retired in 2022; the string is kept for legacy matching.
Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +99
Redirected —
ia_archiver-web.archive.org
A distinct token used by the Wayback Machine's live-web / Save Page Now fetcher.
Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +99
Redirected —
TurnitinBot
A newer TurnitinBot/2.x is reported but not independently confirmed.
Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +99
HTTP 403 the server (HTTP 403)
ViennaTinyBot
Only a bare lowercase identifier appears in aggregator directories.
Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +99
Not probed —

Section F · Reference

Reference

7 terms, 10 questions answered.

The seven words this report uses

Gate 1 · your robots.txt (server-robots-instructions)
What your robots.txt says about that bot's name. Costs your server one request in total: the file is fetched once and read for all 157 bots.
Gate 2 · your server, live (server-crawl-live-test)
What your server answers to one live HEAD request wearing that bot's own user-agent. Headers only — no page content travels.
Allowed / Allowed with exceptions / Not allowed / Not indicated
The four robots.txt verdicts. Allowed with exceptions means the homepage is open but the governing group closes some paths — the normal shape of a well-kept file; it counts as allowed, and the closed paths are tested against your known content pages. Not allowed means the homepage itself is closed. Not indicated means no group names the bot and the file has no catch-all group: by the standard, silence is permission.
Served
A normal answer to the live request — that bot gets your page.
Refused · captcha wall
An HTTP refusal (403 and its neighbours), or a success code that led to a challenge page instead of your homepage. Bots do not solve captchas, so the report counts a wall as a refusal and names it.
Rate-limited
HTTP 429: a statement about how fast our test asked, not about who may crawl you. Left out of every figure.
Not probed
The operator publishes a robots.txt token but no user-agent string, so no honest knock is possible. Judged on your file alone.

Frequently asked questions

What is the server-robots-instructions test?

It reads your robots.txt — the public rulebook every well-behaved bot fetches before anything else — and looks up each bot's name in it. Four answers are possible: Allowed (the governing group closes nothing), Allowed with exceptions (the homepage is open but some paths are closed — the report lists them and says whether any is a page you publish), Not allowed (the homepage itself is closed), and Not indicated (no group names it and no catch-all exists — by the standard, silence means permission). This test costs your server one request in total: the file is fetched once and read for all 157 bots.

What is the server-crawl-live-test?

One real HTTP HEAD request to your homepage per bot, sent with that bot's own User-Agent header — the way the bot itself would knock. HEAD asks for the headers only, so no page content travels and the test costs your server almost nothing. The answer is Allowed (a normal response), Forbidden (an HTTP refusal such as 403, or a captcha wall), or Error (a timeout, a network failure or a broken 5xx answer).

What is html-meta-robots — and why is it not in this report?

html-meta-robots is the <meta name="robots" content="noindex"> tag inside a page's HTML: a third place where a site can tell bots what to do, page by page. This report never downloads your HTML (the live test is header-only by design), so it cannot read that tag — but it does read the tag's HTTP twin, the X-Robots-Tag response header, which arrives with every answer: that is the "Indexing directives in the headers" check in each group. The html-meta-robots of each promoted page is checked in the page SEO/GEO reports, which do read the HTML.

Can the two tests contradict each other?

Yes, in both directions, and the contradiction is one of the most useful findings here. Invited but refused: robots.txt allows a bot, yet the server turns its request away — a firewall or bot wall is overruling the file, possibly without anyone having decided that. Blocked but served: robots.txt forbids a bot, yet the server answers anyway — the block is a polite request that only honest bots obey. The "Do the two tests agree?" check in each group lists every case found.

Does a robots.txt block actually stop a bot?

Only if the bot chooses to obey it. robots.txt is a convention, not an enforcement: the major search engines and AI crawlers respect it scrupulously, but nothing technical prevents a scraper from ignoring the file and requesting your pages anyway. If a block really matters to you, it needs a server-side rule too — which is exactly what the server-crawl-live-test measures.

Why might Googlebot show "Forbidden" here while Google still indexes my site?

Because this test sends Googlebot's name from AkOOre's network, not from Google's. Google, Bing and Apple all recommend that sites verify crawlers by IP address or reverse DNS rather than by name — a server doing that correctly refuses anyone who merely claims to be Googlebot while serving the real one normally. That is why the report treats a refusal as a question to investigate (check Search Console or your server logs), and why the control request exists.

What does "Captcha" mean in the live-test column?

The server answered with a success code but sent the bot to a challenge page — a CAPTCHA or a JavaScript interstitial — instead of your homepage. For a crawler that is a refusal with extra steps: bots do not solve captchas, so the page behind the wall is never read. The report counts it as Forbidden and names the challenge, because a naive reading of the 200 status would have called it allowed.

What does "Not probed" mean?

Some bot operators publish a robots.txt token but never a literal user-agent string (Google-Extended and Applebot-Extended, for example, ride along inside other crawlers' requests). Those bots cannot be impersonated honestly — inventing a string would mean inventing a verdict — so they are judged on your robots.txt alone and left out of the request percentage.

What does "Rate-limited — not measurable" mean?

Your server answered HTTP 429 ("too many requests") — a statement about how fast our test asked, not about whether that bot is welcome. Counting it as a refusal would blame your site for our pace, so those rows are left out of every percentage. The test slows itself down when it sees a 429; Refresh later usually measures them cleanly.

Should I block the AI bots?

Split them in two first. Citation crawlers (OAI-SearchBot, ChatGPT-User, Claude-SearchBot, PerplexityBot…) fetch pages so an assistant can quote and link you — blocking them makes you invisible in answers your customers increasingly read. Training crawlers (GPTBot, CCBot, Meta-ExternalAgent…) collect content to train models — blocking them protects your content and costs you no citations. Blocking training while allowing citation is a common, coherent middle ground; the AI assistants section shows exactly which door each bot is.

Add to your ToDo list

Send feedback