Crawler access — OriginalUniqueArt.com

Section A

Who can read your site

Critical

Guide

This section answers: Can Google, Bing, ChatGPT and the other assistants read my website? Open guide

What to fix first, the verdict for each kind of robot, then why a robot is refused when one is.

Work orders

4 critical fixes, 26 in all.

Guide

This section answers: What do I change, and in what order? Open guide

Fix first — critical 4

Server-crawl-live-test — the live HEAD request (D. Google's own robots — this check scores 0/100)
What to do: Read this one before acting: an ordinary desktop browser was refused too, so your server is not singling out crawlers — it is refusing our test requests wholesale (a rate limit, a geo or IP rule, or a challenge page). Check what your firewall does with unfamiliar visitors before changing any bot rule.

What we measured on your site: 8 forbidden: Gemini-Deep-Research, Google-InspectionTool, GoogleOther, GoogleOther-Image, GoogleOther-Video, Googlebot-Image, Googlebot-Video, Storebot-Google

Why it matters: What your server really answers to one HEAD request wearing that bot's name. A firewall, a CDN rule or a bot-protection product can refuse a crawler your robots.txt welcomes — and robots.txt alone can never reveal it.

Where to see the detail: the D. Google's own robots section shows this check with the data it is based on.
The bots that decide the outcome (D. Google's own robots — this check scores 0/100)
What to do: Fix these before anything else in this report: each one is a whole audience that cannot see the site today. The table below names the exact rule or answer responsible for each.

What we measured on your site: Google-InspectionTool is stopped by the server (HTTP 403); GoogleOther is stopped by a bot wall instead of the page — redirected to a captcha page; Googlebot-Image is stopped by a bot wall instead of…

Why it matters: Not every bot in this group is worth the same. These are the ones whose verdict changes what your business gets: the indexes people actually search and the assistants they actually ask.

Where to see the detail: the D. Google's own robots section shows this check with the data it is based on.
Server-crawl-live-test — the live HEAD request (B. Search engines — this check scores 15/100)
What to do: Read this one before acting: an ordinary desktop browser was refused too, so your server is not singling out crawlers — it is refusing our test requests wholesale (a rate limit, a geo or IP rule, or a challenge page). Check what your firewall does with unfamiliar visitors before changing any bot rule.

What we measured on your site: 17 forbidden: Amzn-SearchBot, Applebot, AspiegelBot, Baiduspider, BingPreview, Googlebot, MicrosoftPreview, MojeekBot and 9 more

Why it matters: What your server really answers to one HEAD request wearing that bot's name. A firewall, a CDN rule or a bot-protection product can refuse a crawler your robots.txt welcomes — and robots.txt alone can never reveal it.

Where to see the detail: the B. Search engines section shows this check with the data it is based on.
The bots that decide the outcome (B. Search engines — this check scores 33/100)
What to do: Fix these before anything else in this report: each one is a whole audience that cannot see the site today. The table below names the exact rule or answer responsible for each.

What we measured on your site: Applebot is stopped by the server (HTTP 403); Baiduspider is stopped by the server (HTTP 403); Googlebot is stopped by a bot wall instead of the page — redirected to a captcha page; YandexBot is stop…

Why it matters: Not every bot in this group is worth the same. These are the ones whose verdict changes what your business gets: the indexes people actually search and the assistants they actually ask.

Where to see the detail: the B. Search engines section shows this check with the data it is based on.

Then improve 4

Server-crawl-live-test — the live HEAD request (C. AI assistants — this check scores 58/100)
What to do: Read this one before acting: an ordinary desktop browser was refused too, so your server is not singling out crawlers — it is refusing our test requests wholesale (a rate limit, a geo or IP rule, or a challenge page). Check what your firewall does with unfamiliar visitors before changing any bot rule.

What we measured on your site: 10 forbidden: AI2Bot, Ai2Bot-Dolma, DeepSeekBot, DigitalOceanGenAICrawler, DuckAssistBot, LINER Bot, PanguBot, SBIntuitionsBot and 2 more

Why it matters: What your server really answers to one HEAD request wearing that bot's name. A firewall, a CDN rule or a bot-protection product can refuse a crawler your robots.txt welcomes — and robots.txt alone can never reveal it.

Where to see the detail: the C. AI assistants section shows this check with the data it is based on.
Server-crawl-live-test — the live HEAD request (E. Social Bots — this check scores 64/100)
What to do: Read this one before acting: an ordinary desktop browser was refused too, so your server is not singling out crawlers — it is refusing our test requests wholesale (a rate limit, a geo or IP rule, or a challenge page). Check what your firewall does with unfamiliar visitors before changing any bot rule.

What we measured on your site: 4 forbidden: Meta-ExternalAds, Meta-ExternalFetcher, Meta-WebIndexer, Quora-Bot

Why it matters: What your server really answers to one HEAD request wearing that bot's name. A firewall, a CDN rule or a bot-protection product can refuse a crawler your robots.txt welcomes — and robots.txt alone can never reveal it.

Where to see the detail: the E. Social Bots section shows this check with the data it is based on.
Exceptions that close content pages (D. Google's own robots — this check scores 70/100)
What to do: Open robots.txt and look at the User-agent group named for each of these bots in the table below: one of its Disallow lines matches a page you publish (/en/orders/photo-to-painting/corporate/, /en/orders/photo-to-painting/family/, /en/orders/photo-to-painting/startup/ and 1 more). Narrow the pattern to the private path you meant, or add an Allow line for the content section — unless hiding those pages from these bots is deliberate.

What we measured on your site: 1 bot(s) have exceptions closing 4 of the 500 known content pages — Googlebot-Image — on /en/orders/photo-to-painting/corporate/, /en/orders/photo-to-painting/family/, /en/orders/photo-to-painting/st…

Why it matters: Whether the paths a robots.txt group closes are housekeeping (admin, cart, search results) or the pages you want found. Each bot's exceptions are tested against this site's known content pages; a closed content page is invisible to that bot whatever the homepage verdict says.

Where to see the detail: the D. Google's own robots section shows this check with the data it is based on.
Exceptions that close content pages (E. Social Bots — this check scores 70/100)
What to do: Open robots.txt and look at the User-agent group named for each of these bots in the table below: one of its Disallow lines matches a page you publish (/en/orders/photo-to-painting/corporate/, /en/orders/photo-to-painting/family/, /en/orders/photo-to-painting/startup/ and 1 more). Narrow the pattern to the private path you meant, or add an Allow line for the content section — unless hiding those pages from these bots is deliberate.

What we measured on your site: 2 bot(s) have exceptions closing 4 of the 500 known content pages — Twitterbot, facebookexternalhit — on /en/orders/photo-to-painting/corporate/, /en/orders/photo-to-painting/family/, /en/orders/phot…

Why it matters: Whether the paths a robots.txt group closes are housekeeping (admin, cart, search results) or the pages you want found. Each bot's exceptions are tested against this site's known content pages; a closed content page is invisible to that bot whatever the homepage verdict says.

Where to see the detail: the E. Social Bots section shows this check with the data it is based on.

Worth knowing 18

Do the two tests agree? (B. Search engines)
What to do with it: A contradiction is worth resolving whichever way you meant it. Invited-but-refused: your firewall overrules your robots.txt — align them, or accept that the file is advertising a welcome the server does not honour. Blocked-but-served: your robots.txt line is a polite request only; a bot that ignores it gets the page, so add a server rule if the block really matters.

What the data says: 17 bot(s) invited by robots.txt but refused by the server — the same ones the live test above names

Why it matters: Whether the server-robots-instructions and the server-crawl-live-test tell the same story. They often do not: robots.txt is a published promise, the live answer is what actually happens, and only the comparison shows which one to believe.

Where to see the detail: the B. Search engines section shows this check with the data it is based on.
Judged on robots.txt alone (B. Search engines)
What to do with it: Nothing to do. Their robots.txt verdict is still exact — it is read from your own file — only the server-side half is unknown.

What the data says: 7 of 27: Applebot-Extended, BingVideoPreview, Yahoo-Blogs, YahooSeeker, Yandex, YandexAdditionalBot, baidu

Why it matters: Some operators publish a robots.txt token but never a literal user-agent string. Those bots cannot be impersonated honestly, so they are judged on your rules only — no live request was made in their name.

Where to see the detail: the B. Search engines section shows this check with the data it is based on.
Do the two tests agree? (C. AI assistants)
What to do with it: A contradiction is worth resolving whichever way you meant it. Invited-but-refused: your firewall overrules your robots.txt — align them, or accept that the file is advertising a welcome the server does not honour. Blocked-but-served: your robots.txt line is a polite request only; a bot that ignores it gets the page, so add a server rule if the block really matters.

What the data says: 10 bot(s) invited by robots.txt but refused by the server — the same ones the live test above names

Why it matters: Whether the server-robots-instructions and the server-crawl-live-test tell the same story. They often do not: robots.txt is a published promise, the live answer is what actually happens, and only the comparison shows which one to believe.

Where to see the detail: the C. AI assistants section shows this check with the data it is based on.
Judged on robots.txt alone (C. AI assistants)
What to do with it: Nothing to do. Their robots.txt verdict is still exact — it is read from your own file — only the server-side half is unknown.

What the data says: 9 of 33: ChatGPT Operator, ChatGPT-User/2.0, Google-Extended, Grok, GrokBot, MistralAI-User, MyCentralAIScraperBot, Perplexity-User and 1 more

Why it matters: Some operators publish a robots.txt token but never a literal user-agent string. Those bots cannot be impersonated honestly, so they are judged on your rules only — no live request was made in their name.

Where to see the detail: the C. AI assistants section shows this check with the data it is based on.
Do the two tests agree? (D. Google's own robots)
What to do with it: A contradiction is worth resolving whichever way you meant it. Invited-but-refused: your firewall overrules your robots.txt — align them, or accept that the file is advertising a welcome the server does not honour. Blocked-but-served: your robots.txt line is a polite request only; a bot that ignores it gets the page, so add a server rule if the block really matters.

What the data says: 8 bot(s) invited by robots.txt but refused by the server — the same ones the live test above names

Why it matters: Whether the server-robots-instructions and the server-crawl-live-test tell the same story. They often do not: robots.txt is a published promise, the live answer is what actually happens, and only the comparison shows which one to believe.

Where to see the detail: the D. Google's own robots section shows this check with the data it is based on.
Judged on robots.txt alone (D. Google's own robots)
What to do with it: Nothing to do. Their robots.txt verdict is still exact — it is read from your own file — only the server-side half is unknown.

What the data says: 2 of 10: Googlebot-Discovery, Googlebot-News

Why it matters: Some operators publish a robots.txt token but never a literal user-agent string. Those bots cannot be impersonated honestly, so they are judged on your rules only — no live request was made in their name.

Where to see the detail: the D. Google's own robots section shows this check with the data it is based on.
Do the two tests agree? (E. Social Bots)
What to do with it: A contradiction is worth resolving whichever way you meant it. Invited-but-refused: your firewall overrules your robots.txt — align them, or accept that the file is advertising a welcome the server does not honour. Blocked-but-served: your robots.txt line is a polite request only; a bot that ignores it gets the page, so add a server rule if the block really matters.

What the data says: 4 bot(s) invited by robots.txt but refused by the server — the same ones the live test above names

Why it matters: Whether the server-robots-instructions and the server-crawl-live-test tell the same story. They often do not: robots.txt is a published promise, the live answer is what actually happens, and only the comparison shows which one to believe.

Where to see the detail: the E. Social Bots section shows this check with the data it is based on.
Server-crawl-live-test — the live HEAD request (E. SEO Tools)
What to do with it: Read this one before acting: an ordinary desktop browser was refused too, so your server is not singling out crawlers — it is refusing our test requests wholesale (a rate limit, a geo or IP rule, or a challenge page). Check what your firewall does with unfamiliar visitors before changing any bot rule.

What the data says: 20 forbidden: AhrefsBot, AudigentAdBot, AwarioRssBot, AwarioSmartBot, Brandwatch, Chrome-Lighthouse, DataForSeoBot, DotBot and 12 more

Why it matters: What your server really answers to one HEAD request wearing that bot's name. A firewall, a CDN rule or a bot-protection product can refuse a crawler your robots.txt welcomes — and robots.txt alone can never reveal it.

Where to see the detail: the E. SEO Tools section shows this check with the data it is based on.
Do the two tests agree? (E. SEO Tools)
What to do with it: A contradiction is worth resolving whichever way you meant it. Invited-but-refused: your firewall overrules your robots.txt — align them, or accept that the file is advertising a welcome the server does not honour. Blocked-but-served: your robots.txt line is a polite request only; a bot that ignores it gets the page, so add a server rule if the block really matters.

What the data says: 20 bot(s) invited by robots.txt but refused by the server — the same ones the live test above names

Why it matters: Whether the server-robots-instructions and the server-crawl-live-test tell the same story. They often do not: robots.txt is a published promise, the live answer is what actually happens, and only the comparison shows which one to believe.

Where to see the detail: the E. SEO Tools section shows this check with the data it is based on.
Judged on robots.txt alone (E. SEO Tools)
What to do with it: Nothing to do. Their robots.txt verdict is still exact — it is read from your own file — only the server-side half is unknown.

What the data says: 1 of 21: NeticleBot

Why it matters: Some operators publish a robots.txt token but never a literal user-agent string. Those bots cannot be impersonated honestly, so they are judged on your rules only — no live request was made in their name.

Where to see the detail: the E. SEO Tools section shows this check with the data it is based on.
Server-crawl-live-test — the live HEAD request (E. Scrapers)
What to do with it: Read this one before acting: an ordinary desktop browser was refused too, so your server is not singling out crawlers — it is refusing our test requests wholesale (a rate limit, a geo or IP rule, or a challenge page). Check what your firewall does with unfamiliar visitors before changing any bot rule.

What the data says: 34 forbidden: Arquivo-web-crawler, BLEXBot, Bytespider, CCBot, Cotoyogi, Crawl4AI, Diffbot, EchoboxBot and 26 more

Why it matters: What your server really answers to one HEAD request wearing that bot's name. A firewall, a CDN rule or a bot-protection product can refuse a crawler your robots.txt welcomes — and robots.txt alone can never reveal it.

Where to see the detail: the E. Scrapers section shows this check with the data it is based on.
Do the two tests agree? (E. Scrapers)
What to do with it: A contradiction is worth resolving whichever way you meant it. Invited-but-refused: your firewall overrules your robots.txt — align them, or accept that the file is advertising a welcome the server does not honour. Blocked-but-served: your robots.txt line is a polite request only; a bot that ignores it gets the page, so add a server rule if the block really matters.

What the data says: 34 bot(s) invited by robots.txt but refused by the server — the same ones the live test above names

Why it matters: Whether the server-robots-instructions and the server-crawl-live-test tell the same story. They often do not: robots.txt is a published promise, the live answer is what actually happens, and only the comparison shows which one to believe.

Where to see the detail: the E. Scrapers section shows this check with the data it is based on.
Judged on robots.txt alone (E. Scrapers)
What to do with it: Nothing to do. Their robots.txt verdict is still exact — it is read from your own file — only the server-side half is unknown.

What the data says: 11 of 46: Amazon Kendra, Bravest, Crawlspace, Echobot Bot, JenkersBot, Poseidon Research Crawler, Seekr, SeekrBot and 3 more

Why it matters: Some operators publish a robots.txt token but never a literal user-agent string. Those bots cannot be impersonated honestly, so they are judged on your rules only — no live request was made in their name.

Where to see the detail: the E. Scrapers section shows this check with the data it is based on.
Server-crawl-live-test — the live HEAD request (E. Cloud Services)
What to do with it: Read this one before acting: an ordinary desktop browser was refused too, so your server is not singling out crawlers — it is refusing our test requests wholesale (a rate limit, a geo or IP rule, or a challenge page). Check what your firewall does with unfamiliar visitors before changing any bot rule.

What the data says: 4 forbidden: AliyunSecBot, Amazonbot, Amzn-User, Google-CloudVertexBot

Why it matters: What your server really answers to one HEAD request wearing that bot's name. A firewall, a CDN rule or a bot-protection product can refuse a crawler your robots.txt welcomes — and robots.txt alone can never reveal it.

Where to see the detail: the E. Cloud Services section shows this check with the data it is based on.
Do the two tests agree? (E. Cloud Services)
What to do with it: A contradiction is worth resolving whichever way you meant it. Invited-but-refused: your firewall overrules your robots.txt — align them, or accept that the file is advertising a welcome the server does not honour. Blocked-but-served: your robots.txt line is a polite request only; a bot that ignores it gets the page, so add a server rule if the block really matters.

What the data says: 4 bot(s) invited by robots.txt but refused by the server — the same ones the live test above names

Why it matters: Whether the server-robots-instructions and the server-crawl-live-test tell the same story. They often do not: robots.txt is a published promise, the live answer is what actually happens, and only the comparison shows which one to believe.

Where to see the detail: the E. Cloud Services section shows this check with the data it is based on.
Server-crawl-live-test — the live HEAD request (E. Other Agents)
What to do with it: Read this one before acting: an ordinary desktop browser was refused too, so your server is not singling out crawlers — it is refusing our test requests wholesale (a rate limit, a geo or IP rule, or a challenge page). Check what your firewall does with unfamiliar visitors before changing any bot rule.

What the data says: 4 forbidden: TurnitinBot, archive.org_bot, ia_archiver, ia_archiver-web.archive.org

Why it matters: What your server really answers to one HEAD request wearing that bot's name. A firewall, a CDN rule or a bot-protection product can refuse a crawler your robots.txt welcomes — and robots.txt alone can never reveal it.

Where to see the detail: the E. Other Agents section shows this check with the data it is based on.
Do the two tests agree? (E. Other Agents)
What to do with it: A contradiction is worth resolving whichever way you meant it. Invited-but-refused: your firewall overrules your robots.txt — align them, or accept that the file is advertising a welcome the server does not honour. Blocked-but-served: your robots.txt line is a polite request only; a bot that ignores it gets the page, so add a server rule if the block really matters.

What the data says: 4 bot(s) invited by robots.txt but refused by the server — the same ones the live test above names

Why it matters: Whether the server-robots-instructions and the server-crawl-live-test tell the same story. They often do not: robots.txt is a published promise, the live answer is what actually happens, and only the comparison shows which one to believe.

Where to see the detail: the E. Other Agents section shows this check with the data it is based on.
Judged on robots.txt alone (E. Other Agents)
What to do with it: Nothing to do. Their robots.txt verdict is still exact — it is read from your own file — only the server-side half is unknown.

What the data says: 1 of 5: ViennaTinyBot

Why it matters: Some operators publish a robots.txt token but never a literal user-agent string. Those bots cannot be impersonated honestly, so they are judged on your rules only — no live request was made in their name.

Where to see the detail: the E. Other Agents section shows this check with the data it is based on.

Help The whole report as a to-do list: Almost every line is one line in a text file or one rule in a firewall — the cheapest work in SEO, and the only kind no amount of content can replace. Read more

Help How it is sorted: Fix first: a group that decides whether you are findable is shut out — worst first. Then improve: partial restrictions, check they are the ones you meant. Worth knowing: not faults, including doors you may close on purpose. Read more

Who can actually read your site

101 of 157 bots turned away.

Guide

This section answers: Of the 157 named bots that knock, which ones reach a page? Open guide

Words used

Robot rules file: — a small file on your site that tells robots which pages they may read.
Robot name: — the name a robot gives when it visits your site.
Control request

157 bots tested 8 groups 126 probed live with their own user-agent Nov. 2, 2026, 2:04 a.m. last probe

Your robots.txt says yes to all 157. Your server says no to 101.

157 bots knock originaluniqueart.com

Gate 1 Your robots.txt

157 — allowed with exceptions

32 of them have a group of their own in your file, 125 ride on the catch-all group (User-agent: *) and 0 are named by no rule. 157 of them are allowed with exceptions, and for 3 the exceptions close known content pages: /en/orders/photo-to-painting/corporate/, /en/orders/photo-to-painting/family/, /en/orders/photo-to-painting/startup/, /en/orders/photo_to_restyled_masterpiece/.

Gate 2 Your server, live

25 10 91 31

ServedCaptcha wallRefusedNot probed

25 Read your site ChatGPT-UserClaude-CodeClaude-SearchBotClaude-UserClaude-WebClaudeBotGPTBotMistralAI-User/1.0OAI-SearchBotPerplexity-User/1.0 + 15 more

101 Turned away by the server AI2Bot HTTP 403Ai2Bot-Dolma HTTP 403DeepSeekBot HTTP 403DigitalOceanGenAICrawler HTTP 403DuckAssistBot HTTP 403LINER Bot HTTP 403PanguBot HTTP 403SBIntuitionsBot HTTP 403cohere-training-data-crawler HTTP 403img2dataset HTTP 403 + 91 more

31 Judged on the file alone ChatGPT OperatorChatGPT-User/2.0Google-ExtendedGrokGrokBotMistralAI-UserMyCentralAIScraperBotPerplexity-Userquillbot.comGooglebot-Discovery + 21 more

Their operator publishes no user-agent string, so no honest knock is possible.

Help How it was measured: Each bot is tested twice, the way it really arrives: what your robots.txt says about its name, then what your server answers to one live request wearing it. Only the second is measurable from outside. A bot whose operator publishes no user-agent string is judged on your file alone — inventing a string would mean inventing a verdict. Read more

Help Reading the bars: Green = the bot gets your page; hatched green = allowed, but the file closes some paths. Dark or red = refused; hatched = a captcha wall, which bots cannot solve. Grey = not measurable. Read more

Instrument check

HTTP 403 — a plain browser was refused too

Sent as Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/140.0.0.0 Safari/537.36

Careful The refusals are not about crawlers: This server turns away our test requests whatever they call themselves (a rate limit, an IP rule, a challenge page). Resolve that first — every bot reading is unreliable until you do. Read more

The figures and the access standing

Critical

64/100, 4 critical checks.

Guide

This section answers: How much of the traffic that matters gets through? Open guide

Words used

Scored check
Welcome

64/100 Critical

05080100

Critical: 4
Needs work: 4
Good: 6

157/157 Allowed in robots.txt

25/126 Requests allowed

10/27 Search engines welcome

23/33 AI answer engines welcome

101 Requests forbidden

Help The score: Average of the 14 scored check points (the 40 include informational ones). Googlebot, Bingbot, GPTBot, ClaudeBot and their peers weigh far more than the long tail.

Critical: Now. It is costing money or visibility today.
Fix soon: This week, before anything new.
Good: Fine as it is; keep going.
Worth knowing: Nothing to fix; good to know.

Info is a check point with no score. Read more

Help What the figures count: Requests allowed: live requests answered normally, of those measurable. Welcome: bots that pass both gates. Forbidden while robots.txt invites them in is usually a firewall rule nobody remembers writing. Read more

Every group, and which door it is behind

Lowest scored group: Google Bots, 42%.

Guide

This section answers: Which group should I open next? Open guide

Every bot group, with how many bots got each outcome
Group	Score	Served	Captcha wall	Refused	Not probed
Scored — being shut out here costs you
A. Search Engines	49% Fix soon	3	1	16	7
B. AI Bots	81% Good	14	·	10	9
C. Google Bots	42% Fix soon	·	5	3	2
D. Social Bots	84% Good	7	·	4	·
Measured, never marked wrong
E. SEO Tools	not scored	·	·	20	1
F. Scrapers	not scored	1	1	33	11
G. Cloud Services	not scored	·	·	4	·
H. Other Agents	not scored	·	3	1	1

Help Reading the grid: One row per group, one column per outcome. Darker = a bigger share of the group. Scored groups use strong colour; the others stay pale on purpose. Click a group to open it. Read more

Help Why only four are scored: Shut out of search engines, AI bots, social bots or the Google family, you lose traffic, citations or link previews. Blocking a scraper is a legitimate decision about your own content, so those groups are shown and never marked wrong. Read more

The posture we measured

101 contradictions between file and server.

Guide

This section answers: Is this door policy a decision, or an accident? Open guide

Words used

Posture
Invited, then refused
Blocked, then served

Strategy

Hard wall

robots.txt welcomes the bots, but the server turns most requests away (a firewall, CDN rule or bot challenge). The published rulebook and the door policy disagree — the door wins.

What it buys you

Strong, enforced protection: bots that ignore robots.txt are stopped anyway.
Legitimate if crawler verification by IP or reverse DNS lets the real search engines through a door that name-claimers cannot use.

What it costs you

If the wall does NOT allowlist real crawlers by IP, search engines and AI assistants are locked out with everyone else — invisibility, not protection.
The welcoming robots.txt is misleading — it advertises an access the server does not grant.

Where your two rulebooks contradict each other

101 invited, then refused

robots.txt welcomes AI2Bot, Ai2Bot-Dolma, DeepSeekBot, DigitalOceanGenAICrawler, DuckAssistBot, LINER Bot and 95 more (among others), but the server turns them away. The server wins — your file is advertising a welcome it does not honour.

Careful Before you change a firewall rule: The probe sends each bot's user-agent from AkOOre's network, not from Google's or OpenAI's. A server that verifies crawlers by IP or reverse DNS correctly refuses a request that merely claims to be Googlebot. Confirm in Search Console or your server logs first. Read more

Help Reading the posture: Not a finding but a reading of the door policy the site actually runs. Every posture is legitimate — the point is to make it a conscious one. Each group's “Do the two tests agree?” check names every contradiction. Read more

Section B · Scored group

Search engines

Critical

3 of 27 bots read your site.

Guide

This section answers: Can Google and Bing read my pages? Open guide

Words used

Gate 1
Gate 2
Captcha wall

Measured Nov. 2, 2026, 2:04 a.m.

Help Who these bots are: The crawlers that build the search results people click. If one of them cannot fetch your pages, you are not ranking badly — you are not in that index at all, and no amount of content or link work changes it. Each bot is tested twice: what your robots.txt permits for its name, then what your server actually answers to a request arriving under that name. Read more

1/3 scored points pass. Weakest point: Server-crawl-live-test — the live HEAD request (15%).

Your robots.txt says yes to all 27. Your server says no to 17.

27 bots knock A. Search Engines

Gate 1 Your robots.txt

27 — allowed with exceptions

2 of them have a group of their own in your file, 25 ride on the catch-all group (User-agent: *) and 0 are named by no rule. 27 of them are allowed with exceptions — the closed paths are housekeeping only, no known content page is among them.

Gate 2 Your server, live

3 1 16 7

ServedCaptcha wallRefusedNot probed

3 Read your site BingbotDuckDuckBotadidxbot

17 Turned away by the server Amzn-SearchBot HTTP 403Applebot HTTP 403AspiegelBot HTTP 403Baiduspider HTTP 403BingPreview HTTP 403MicrosoftPreview HTTP 403MojeekBot HTTP 403PetalBot HTTP 403SeznamBot HTTP 403SeznamHomepageCrawler HTTP 403 + 7 more

7 Judged on the file alone Applebot-ExtendedBingVideoPreviewYahoo-BlogsYahooSeekerYandexYandexAdditionalBotbaidu

Their operator publishes no user-agent string, so no honest knock is possible.

Help Reading the doors: Gate 1 is your robots.txt, gate 2 your server's live answer, then the outcome each bot got. Hatched red = a captcha wall; grey = not measurable. Read more

Help Why this group is scored: Shut out here, you lose traffic, citations or link previews — so a refusal is painted red and costs points. Read more

What the measurement means, point by point

Good Server-robots-instructions — what robots.txt permits 100%

CheckedThe first thing every well-behaved bot reads. A Disallow rule on the homepage stops it before it ever requests a page, so nothing further down can compensate. A group that closes some paths but not the homepage is 'allowed with exceptions' — it counts as allowed, and the exceptions are tested against your known content pages below. A bot no rule names is allowed — silence is permission — and shown as 'not indicated'.

Foundall 27 are allowed by robots.txt; 27 of them are allowed with exceptions — the closed paths are housekeeping only, no known content page is among them.

Critical Server-crawl-live-test — the live HEAD request 15%

CheckedWhat your server really answers to one HEAD request wearing that bot's name. A firewall, a CDN rule or a bot-protection product can refuse a crawler your robots.txt welcomes — and robots.txt alone can never reveal it.

Found17 forbidden: Amzn-SearchBot, Applebot, AspiegelBot, Baiduspider, BingPreview, Googlebot, MicrosoftPreview, MojeekBot and 9 more

Do thisRead this one before acting: an ordinary desktop browser was refused too, so your server is not singling out crawlers — it is refusing our test requests wholesale (a rate limit, a geo or IP rule, or a challenge page). Check what your firewall does with unfamiliar visitors before changing any bot rule.

Worth knowing Indexing directives in the headers

CheckedThe X-Robots-Tag response header of the answers your server actually served. A noindex here lets the bot in and then tells it to throw the page away — the most expensive way to be invisible, because it looks fine in every other check. Only served answers are read: a wall's or an error page's header is not yours.

Foundno X-Robots-Tag header in any served answer

Worth knowing Do the two tests agree?

CheckedWhether the server-robots-instructions and the server-crawl-live-test tell the same story. They often do not: robots.txt is a published promise, the live answer is what actually happens, and only the comparison shows which one to believe.

Found17 bot(s) invited by robots.txt but refused by the server — the same ones the live test above names

Worth knowingA contradiction is worth resolving whichever way you meant it. Invited-but-refused: your firewall overrules your robots.txt — align them, or accept that the file is advertising a welcome the server does not honour. Blocked-but-served: your robots.txt line is a polite request only; a bot that ignores it gets the page, so add a server rule if the block really matters.

Critical The bots that decide the outcome 33%

CheckedNot every bot in this group is worth the same. These are the ones whose verdict changes what your business gets: the indexes people actually search and the assistants they actually ask.

FoundApplebot is stopped by the server (HTTP 403); Baiduspider is stopped by the server (HTTP 403); Googlebot is stopped by a bot wall instead of the page — redirected to a captcha page; YandexBot is stopped by the server (HTTP 403)

Do thisFix these before anything else in this report: each one is a whole audience that cannot see the site today. The table below names the exact rule or answer responsible for each.

Worth knowing Judged on robots.txt alone

CheckedSome operators publish a robots.txt token but never a literal user-agent string. Those bots cannot be impersonated honestly, so they are judged on your rules only — no live request was made in their name.

Found7 of 27: Applebot-Extended, BingVideoPreview, Yahoo-Blogs, YahooSeeker, Yandex, YandexAdditionalBot, baidu

Worth knowingNothing to do. Their robots.txt verdict is still exact — it is read from your own file — only the server-side half is unknown.

Help Reading a point: Each point says what was checked, what was found on your site, and what to do. Faults come first; Info points are statements, not passes or fails. Read more

The A. Search Engines group, bot by bot

✗

Amzn-SearchBot

robots.txt: Allowed with exceptions · your server: HTTP 403 · the server (HTTP 403)
✗

Applebot

robots.txt: Allowed with exceptions · your server: HTTP 403 · the server (HTTP 403)
!

Applebot-Extended

robots.txt: Allowed with exceptions · your server: Not probed
✗

AspiegelBot

robots.txt: Allowed with exceptions · your server: HTTP 403 · the server (HTTP 403)
✗

Baiduspider

robots.txt: Allowed with exceptions · your server: HTTP 403 · the server (HTTP 403)
✗

BingPreview

robots.txt: Allowed with exceptions · your server: HTTP 403 · the server (HTTP 403)
!

BingVideoPreview

robots.txt: Allowed with exceptions · your server: Not probed
✓

Bingbot

robots.txt: Allowed with exceptions · your server: Redirected
✓

DuckDuckBot

robots.txt: Allowed with exceptions · your server: Redirected
✗

Googlebot

robots.txt: Allowed with exceptions · your server: Captcha · a bot wall instead of the page — redirected to a captcha page
✗

MicrosoftPreview

robots.txt: Allowed with exceptions · your server: HTTP 403 · the server (HTTP 403)
✗

MojeekBot

robots.txt: Allowed with exceptions · your server: HTTP 403 · the server (HTTP 403)
✗

PetalBot

robots.txt: Allowed with exceptions · your server: HTTP 403 · the server (HTTP 403)
✗

SeznamBot

robots.txt: Allowed with exceptions · your server: HTTP 403 · the server (HTTP 403)
✗

SeznamHomepageCrawler

robots.txt: Allowed with exceptions · your server: HTTP 403 · the server (HTTP 403)
✗

Slurp

robots.txt: Allowed with exceptions · your server: HTTP 403 · the server (HTTP 403)
✗

Teoma

robots.txt: Allowed with exceptions · your server: HTTP 403 · the server (HTTP 403)
!

Yahoo-Blogs

robots.txt: Allowed with exceptions · your server: Not probed
✗

Yahoo-FeedSeeker

robots.txt: Allowed with exceptions · your server: HTTP 403 · the server (HTTP 403)
✗

Yahoo-MMCrawler

robots.txt: Allowed with exceptions · your server: HTTP 403 · the server (HTTP 403)
!

YahooSeeker

robots.txt: Allowed with exceptions · your server: Not probed
!

Yandex

robots.txt: Allowed with exceptions · your server: Not probed
✗

YandexAdditional

robots.txt: Allowed with exceptions · your server: HTTP 403 · the server (HTTP 403)
!

YandexAdditionalBot

robots.txt: Allowed with exceptions · your server: Not probed
✗

YandexBot

robots.txt: Allowed with exceptions · your server: HTTP 403 · the server (HTTP 403)
✓

adidxbot

robots.txt: Allowed with exceptions · your server: Redirected
!

baidu

robots.txt: Allowed with exceptions · your server: Not probed

See the roster table

Every bot of the A. Search Engines group, with its two verdicts
Bot	server-robots-instructions	server-crawl-live-test	What stopped it
Amzn-SearchBot Feeds Amazon, Alexa and Rufus search.	Allowed with exceptions by the catch-all group closed: /media-resize/, /admin/, /*/admin/ +96	HTTP 403	the server (HTTP 403)
Applebot The device, OS and WebKit fields vary between Apple's crawlers.	Allowed with exceptions by the catch-all group closed: /media-resize/, /admin/, /*/admin/ +96	HTTP 403	the server (HTTP 403)
Applebot-Extended A robots.txt token with no distinct user-agent: it governs whether Apple Intelligence may train on content Applebot already fetched.	Allowed with exceptions by `User-agent: applebot-extended` closed: /admin/, /*/admin/, /manage/ +95	Not probed	—
AspiegelBot alias — same bot as another row The legacy name, superseded by PetalBot.	Allowed with exceptions by the catch-all group closed: /media-resize/, /admin/, /*/admin/ +96	HTTP 403	the server (HTTP 403)
Baiduspider	Allowed with exceptions by the catch-all group closed: /media-resize/, /admin/, /*/admin/ +96	HTTP 403	the server (HTTP 403)
BingPreview	Allowed with exceptions by the catch-all group closed: /media-resize/, /admin/, /*/admin/ +96	HTTP 403	the server (HTTP 403)
BingVideoPreview Bing confirms this bot exists but publishes no literal user-agent string, so only its robots.txt token is checked.	Allowed with exceptions by the catch-all group closed: /media-resize/, /admin/, /*/admin/ +96	Not probed	—
Bingbot	Allowed with exceptions by the catch-all group closed: /media-resize/, /admin/, /*/admin/ +96	Redirected	—
DuckDuckBot	Allowed with exceptions by the catch-all group closed: /media-resize/, /admin/, /*/admin/ +96	Redirected	—
Googlebot The mobile variant wraps a Chrome/Android string around the same (compatible; Googlebot/2.1; ...) suffix.	Allowed with exceptions by `User-agent: googlebot` closed: /admin/, /*/admin/, /manage/ +95	Captcha	a bot wall instead of the page — redirected to a captcha page
MicrosoftPreview	Allowed with exceptions by the catch-all group closed: /media-resize/, /admin/, /*/admin/ +96	HTTP 403	the server (HTTP 403)
MojeekBot The version number increments over time (0.2 to 0.11+ seen).	Allowed with exceptions by the catch-all group closed: /media-resize/, /admin/, /*/admin/ +96	HTTP 403	the server (HTTP 403)
PetalBot Huawei Petal Search. The mobile variant prefixes Android and AppleWebKit fields.	Allowed with exceptions by the catch-all group closed: /media-resize/, /admin/, /*/admin/ +96	HTTP 403	the server (HTTP 403)
SeznamBot	Allowed with exceptions by the catch-all group closed: /media-resize/, /admin/, /*/admin/ +96	HTTP 403	the server (HTTP 403)
SeznamHomepageCrawler	Allowed with exceptions by the catch-all group closed: /media-resize/, /admin/, /*/admin/ +96	HTTP 403	the server (HTTP 403)
Slurp	Allowed with exceptions by the catch-all group closed: /media-resize/, /admin/, /*/admin/ +96	HTTP 403	the server (HTTP 403)
Teoma	Allowed with exceptions by the catch-all group closed: /media-resize/, /admin/, /*/admin/ +96	HTTP 403	the server (HTTP 403)
Yahoo-Blogs Legacy, listed only by unofficial aggregators.	Allowed with exceptions by the catch-all group closed: /media-resize/, /admin/, /*/admin/ +96	Not probed	—
Yahoo-FeedSeeker Historical form, not verified as current.	Allowed with exceptions by the catch-all group closed: /media-resize/, /admin/, /*/admin/ +96	HTTP 403	the server (HTTP 403)
Yahoo-MMCrawler Historical form, not verified as current.	Allowed with exceptions by the catch-all group closed: /media-resize/, /admin/, /*/admin/ +96	HTTP 403	the server (HTTP 403)
YahooSeeker Legacy: Yahoo Search is Bing-powered now and no current documentation publishes a string.	Allowed with exceptions by the catch-all group closed: /media-resize/, /admin/, /*/admin/ +96	Not probed	—
Yandex alias — same bot as another row A robots.txt control token matching any Yandex* bot, not a distinct request user-agent.	Allowed with exceptions by the catch-all group closed: /media-resize/, /admin/, /*/admin/ +96	Not probed	—
YandexAdditional Mostly a robots.txt control token; this string comes from secondary sources, not from Yandex's own documentation.	Allowed with exceptions by the catch-all group closed: /media-resize/, /admin/, /*/admin/ +96	HTTP 403	the server (HTTP 403)
YandexAdditionalBot alias — same bot as another row Same bot and same token as YandexAdditional.	Allowed with exceptions by the catch-all group closed: /media-resize/, /admin/, /*/admin/ +96	Not probed	—
YandexBot	Allowed with exceptions by the catch-all group closed: /media-resize/, /admin/, /*/admin/ +96	HTTP 403	the server (HTTP 403)
adidxbot Bing's ad-landing-page quality crawler.	Allowed with exceptions by the catch-all group closed: /media-resize/, /admin/, /*/admin/ +96	Redirected	—
baidu alias — same bot as another row The same bot as Baiduspider, not a separate name or string.	Allowed with exceptions by the catch-all group closed: /media-resize/, /admin/, /*/admin/ +96	Not probed	—

Help Reading the roster: One row per bot: the file's verdict, then the server's. Hover a bot's name for the user-agent we sent, and a verdict for its full detail. The last column says what actually stopped it. Read more

Section C · Scored group

AI assistants

Good

14 of 33 bots read your site.

Guide

This section answers: Can ChatGPT, Perplexity, Claude and Gemini read my pages? Open guide

Words used

Gate 1
Gate 2
Captcha wall

Measured Nov. 2, 2026, 2:04 a.m.

Help Who these bots are: The crawlers behind AI answers — ChatGPT, Claude, Perplexity, Gemini, Copilot. A growing share of buyers asks an assistant instead of searching, and an assistant can only cite pages its crawler was allowed to read. Blocking these is sometimes a deliberate content-protection choice; this section shows exactly which doors are closed so the choice is a conscious one. Read more

2/3 scored points pass. Weakest point: Server-crawl-live-test — the live HEAD request (58%).

Your robots.txt says yes to all 33. Your server says no to 10.

33 bots knock B. AI Bots

Gate 1 Your robots.txt

33 — allowed with exceptions

18 of them have a group of their own in your file, 15 ride on the catch-all group (User-agent: *) and 0 are named by no rule. 33 of them are allowed with exceptions — the closed paths are housekeeping only, no known content page is among them.

Gate 2 Your server, live

14 10 9

ServedRefusedNot probed

14 Read your site ChatGPT-UserClaude-CodeClaude-SearchBotClaude-UserClaude-WebClaudeBotGPTBotMistralAI-User/1.0OAI-SearchBotPerplexity-User/1.0 + 4 more

10 Turned away by the server AI2Bot HTTP 403Ai2Bot-Dolma HTTP 403DeepSeekBot HTTP 403DigitalOceanGenAICrawler HTTP 403DuckAssistBot HTTP 403LINER Bot HTTP 403PanguBot HTTP 403SBIntuitionsBot HTTP 403cohere-training-data-crawler HTTP 403img2dataset HTTP 403

9 Judged on the file alone ChatGPT OperatorChatGPT-User/2.0Google-ExtendedGrokGrokBotMistralAI-UserMyCentralAIScraperBotPerplexity-Userquillbot.com

Their operator publishes no user-agent string, so no honest knock is possible.

Help Reading the doors: Gate 1 is your robots.txt, gate 2 your server's live answer, then the outcome each bot got. Hatched red = a captcha wall; grey = not measurable. Read more

Help Why this group is scored: Shut out here, you lose traffic, citations or link previews — so a refusal is painted red and costs points. Read more

What the measurement means, point by point

Good Server-robots-instructions — what robots.txt permits 100%

CheckedThe first thing every well-behaved bot reads. A Disallow rule on the homepage stops it before it ever requests a page, so nothing further down can compensate. A group that closes some paths but not the homepage is 'allowed with exceptions' — it counts as allowed, and the exceptions are tested against your known content pages below. A bot no rule names is allowed — silence is permission — and shown as 'not indicated'.

Foundall 33 are allowed by robots.txt; 33 of them are allowed with exceptions — the closed paths are housekeeping only, no known content page is among them.

Fix soon Server-crawl-live-test — the live HEAD request 58%

CheckedWhat your server really answers to one HEAD request wearing that bot's name. A firewall, a CDN rule or a bot-protection product can refuse a crawler your robots.txt welcomes — and robots.txt alone can never reveal it.

Found10 forbidden: AI2Bot, Ai2Bot-Dolma, DeepSeekBot, DigitalOceanGenAICrawler, DuckAssistBot, LINER Bot, PanguBot, SBIntuitionsBot and 2 more

Do thisRead this one before acting: an ordinary desktop browser was refused too, so your server is not singling out crawlers — it is refusing our test requests wholesale (a rate limit, a geo or IP rule, or a challenge page). Check what your firewall does with unfamiliar visitors before changing any bot rule.

Worth knowing Indexing directives in the headers

CheckedThe X-Robots-Tag response header of the answers your server actually served. A noindex here lets the bot in and then tells it to throw the page away — the most expensive way to be invisible, because it looks fine in every other check. Only served answers are read: a wall's or an error page's header is not yours.

Foundno X-Robots-Tag header in any served answer

Worth knowing Do the two tests agree?

CheckedWhether the server-robots-instructions and the server-crawl-live-test tell the same story. They often do not: robots.txt is a published promise, the live answer is what actually happens, and only the comparison shows which one to believe.

Found10 bot(s) invited by robots.txt but refused by the server — the same ones the live test above names

Worth knowingA contradiction is worth resolving whichever way you meant it. Invited-but-refused: your firewall overrules your robots.txt — align them, or accept that the file is advertising a welcome the server does not honour. Blocked-but-served: your robots.txt line is a polite request only; a bot that ignores it gets the page, so add a server rule if the block really matters.

Good The bots that decide the outcome 86%

CheckedNot every bot in this group is worth the same. These are the ones whose verdict changes what your business gets: the indexes people actually search and the assistants they actually ask.

FoundDuckAssistBot is stopped by the server (HTTP 403)

Worth knowing Judged on robots.txt alone

CheckedSome operators publish a robots.txt token but never a literal user-agent string. Those bots cannot be impersonated honestly, so they are judged on your rules only — no live request was made in their name.

Found9 of 33: ChatGPT Operator, ChatGPT-User/2.0, Google-Extended, Grok, GrokBot, MistralAI-User, MyCentralAIScraperBot, Perplexity-User and 1 more

Worth knowingNothing to do. Their robots.txt verdict is still exact — it is read from your own file — only the server-side half is unknown.

Help Reading a point: Each point says what was checked, what was found on your site, and what to do. Faults come first; Info points are statements, not passes or fails. Read more

The B. AI Bots group, bot by bot

✗

AI2Bot

robots.txt: Allowed with exceptions · your server: HTTP 403 · the server (HTTP 403)
✗

Ai2Bot-Dolma

robots.txt: Allowed with exceptions · your server: HTTP 403 · the server (HTTP 403)
!

ChatGPT Operator

robots.txt: Allowed with exceptions · your server: Not probed
✓

ChatGPT-User

robots.txt: Allowed with exceptions · your server: Redirected
!

ChatGPT-User/2.0

robots.txt: Allowed with exceptions · your server: Not probed
✓

Claude-Code

robots.txt: Allowed with exceptions · your server: Redirected
✓

Claude-SearchBot

robots.txt: Allowed with exceptions · your server: Redirected
✓

Claude-User

robots.txt: Allowed with exceptions · your server: Redirected
✓

Claude-Web

robots.txt: Allowed with exceptions · your server: Redirected
✓

ClaudeBot

robots.txt: Allowed with exceptions · your server: Redirected
✗

DeepSeekBot

robots.txt: Allowed with exceptions · your server: HTTP 403 · the server (HTTP 403)
✗

DigitalOceanGenAICrawler

robots.txt: Allowed with exceptions · your server: HTTP 403 · the server (HTTP 403)
✗

DuckAssistBot

robots.txt: Allowed with exceptions · your server: HTTP 403 · the server (HTTP 403)
✓

GPTBot

robots.txt: Allowed with exceptions · your server: Redirected
!

Google-Extended

robots.txt: Allowed with exceptions · your server: Not probed
!

Grok

robots.txt: Allowed with exceptions · your server: Not probed
!

GrokBot

robots.txt: Allowed with exceptions · your server: Not probed
✗

LINER Bot

robots.txt: Allowed with exceptions · your server: HTTP 403 · the server (HTTP 403)
!

MistralAI-User

robots.txt: Allowed with exceptions · your server: Not probed
✓

MistralAI-User/1.0

robots.txt: Allowed with exceptions · your server: Redirected
!

MyCentralAIScraperBot

robots.txt: Allowed with exceptions · your server: Not probed
✓

OAI-SearchBot

robots.txt: Allowed with exceptions · your server: Redirected
✗

PanguBot

robots.txt: Allowed with exceptions · your server: HTTP 403 · the server (HTTP 403)
!

Perplexity-User

robots.txt: Allowed with exceptions · your server: Not probed
✓

Perplexity-User/1.0

robots.txt: Allowed with exceptions · your server: Redirected
✓

PerplexityBot

robots.txt: Allowed with exceptions · your server: Redirected
✗

SBIntuitionsBot

robots.txt: Allowed with exceptions · your server: HTTP 403 · the server (HTTP 403)
✓

YouBot

robots.txt: Allowed with exceptions · your server: Redirected
✓

anthropic-ai

robots.txt: Allowed with exceptions · your server: Redirected
✓

cohere-ai

robots.txt: Allowed with exceptions · your server: Redirected
✗

cohere-training-data-crawler

robots.txt: Allowed with exceptions · your server: HTTP 403 · the server (HTTP 403)
✗

img2dataset

robots.txt: Allowed with exceptions · your server: HTTP 403 · the server (HTTP 403)
!

quillbot.com

robots.txt: Allowed with exceptions · your server: Not probed

See the roster table

Every bot of the B. AI Bots group, with its two verdicts
Bot	server-robots-instructions	server-crawl-live-test	What stopped it
AI2Bot A bare token.	Allowed with exceptions by the catch-all group closed: /media-resize/, /admin/, /*/admin/ +96	HTTP 403	the server (HTTP 403)
Ai2Bot-Dolma	Allowed with exceptions by the catch-all group closed: /media-resize/, /admin/, /*/admin/ +96	HTTP 403	the server (HTTP 403)
ChatGPT Operator alias — same bot as another row Folded into ChatGPT agent in 2025. It sends no distinctive user-agent -- it uses an ordinary Chrome desktop string and self-identifies with a Signature-Agent: https://chatgpt.com header, so it cannot be impersonated by user-agent alone.	Allowed with exceptions by `User-agent: chatgpt-user` closed: /admin/, /*/admin/, /manage/ +95	Not probed	—
ChatGPT-User OpenAI's live fetch when a ChatGPT user asks about your page.	Allowed with exceptions by `User-agent: chatgpt-user` closed: /admin/, /*/admin/, /manage/ +95	Redirected	—
ChatGPT-User/2.0 alias — same bot as another row OpenAI's published version is 1.0; there is no evidence of a 2.0, so only the token is checked.	Allowed with exceptions by `User-agent: chatgpt-user` closed: /admin/, /*/admin/, /manage/ +95	Not probed	—
Claude-Code alias — same bot as another row Not a separately documented crawler: the Claude Code CLI's own fetches ride the Claude-User pattern.	Allowed with exceptions by the catch-all group closed: /media-resize/, /admin/, /*/admin/ +96	Redirected	—
Claude-SearchBot	Allowed with exceptions by `User-agent: claude-searchbot` closed: /admin/, /*/admin/, /manage/ +95	Redirected	—
Claude-User	Allowed with exceptions by `User-agent: claude-user` closed: /admin/, /*/admin/, /manage/ +95	Redirected	—
Claude-Web alias — same bot as another row Deprecated, replaced by ClaudeBot / Claude-User / Claude-SearchBot. The string is historical.	Allowed with exceptions by `User-agent: claude-web` closed: /admin/, /*/admin/, /manage/ +95	Redirected	—
ClaudeBot	Allowed with exceptions by `User-agent: claudebot` closed: /admin/, /*/admin/, /manage/ +95	Redirected	—
DeepSeekBot	Allowed with exceptions by the catch-all group closed: /media-resize/, /admin/, /*/admin/ +96	HTTP 403	the server (HTTP 403)
DigitalOceanGenAICrawler Names conflict: DigitalOcean's current docs say DigitalOceanGradientAICrawler/1.0, a widely cited third-party report says DigitalOceanGenAICrawler/1.0.	Allowed with exceptions by the catch-all group closed: /media-resize/, /admin/, /*/admin/ +96	HTTP 403	the server (HTTP 403)
DuckAssistBot	Allowed with exceptions by the catch-all group closed: /media-resize/, /admin/, /*/admin/ +96	HTTP 403	the server (HTTP 403)
GPTBot	Allowed with exceptions by `User-agent: gptbot` closed: /admin/, /*/admin/, /manage/ +95	Redirected	—
Google-Extended A robots.txt token only: the crawl happens under the standard Googlebot or GoogleOther user-agent. It governs whether your pages may ground Gemini answers and AI Overviews.	Allowed with exceptions by `User-agent: google-extended` closed: /admin/, /*/admin/, /manage/ +95	Not probed	—
Grok xAI publishes no crawler documentation page; third-party trackers disagree (Grok, xAI-Grok/1.0, GrokBot/1.0), so no string is sent.	Allowed with exceptions by the catch-all group closed: /media-resize/, /admin/, /*/admin/ +96	Not probed	—
GrokBot alias — same bot as another row Same caveat as Grok: no official documentation.	Allowed with exceptions by the catch-all group closed: /media-resize/, /admin/, /*/admin/ +96	Not probed	—
LINER Bot	Allowed with exceptions by the catch-all group closed: /media-resize/, /admin/, /*/admin/ +96	HTTP 403	the server (HTTP 403)
MistralAI-User alias — same bot as another row 1.0 is the only published version.	Allowed with exceptions by `User-agent: mistralai-user` closed: /admin/, /*/admin/, /manage/ +95	Not probed	—
MistralAI-User/1.0	Allowed with exceptions by `User-agent: mistralai-user` closed: /admin/, /*/admin/, /manage/ +95	Redirected	—
MyCentralAIScraperBot The token circulates in blocklists; the operator and the user-agent string are undocumented.	Allowed with exceptions by the catch-all group closed: /media-resize/, /admin/, /*/admin/ +96	Not probed	—
OAI-SearchBot Builds the index behind ChatGPT search results.	Allowed with exceptions by `User-agent: oai-searchbot` closed: /admin/, /*/admin/, /manage/ +95	Redirected	—
PanguBot Huawei's Pangu LLM crawler.	Allowed with exceptions by the catch-all group closed: /media-resize/, /admin/, /*/admin/ +96	HTTP 403	the server (HTTP 403)
Perplexity-User alias — same bot as another row Same bot as Perplexity-User/1.0.	Allowed with exceptions by `User-agent: perplexity-user` closed: /admin/, /*/admin/, /manage/ +95	Not probed	—
Perplexity-User/1.0	Allowed with exceptions by `User-agent: perplexity-user` closed: /admin/, /*/admin/, /manage/ +95	Redirected	—
PerplexityBot	Allowed with exceptions by `User-agent: perplexitybot` closed: /admin/, /*/admin/, /manage/ +95	Redirected	—
SBIntuitionsBot	Allowed with exceptions by the catch-all group closed: /media-resize/, /admin/, /*/admin/ +96	HTTP 403	the server (HTTP 403)
YouBot	Allowed with exceptions by `User-agent: youbot` closed: /admin/, /*/admin/, /manage/ +95	Redirected	—
anthropic-ai alias — same bot as another row A deprecated legacy token, superseded by ClaudeBot.	Allowed with exceptions by `User-agent: anthropic-ai` closed: /admin/, /*/admin/, /manage/ +95	Redirected	—
cohere-ai A bare token; no Mozilla-wrapped form is documented.	Allowed with exceptions by `User-agent: cohere-ai` closed: /admin/, /*/admin/, /manage/ +95	Redirected	—
cohere-training-data-crawler	Allowed with exceptions by the catch-all group closed: /media-resize/, /admin/, /*/admin/ +96	HTTP 403	the server (HTTP 403)
img2dataset An open-source tool run by many independent operators.	Allowed with exceptions by the catch-all group closed: /media-resize/, /admin/, /*/admin/ +96	HTTP 403	the server (HTTP 403)
quillbot.com The robots.txt token is confirmed; no literal user-agent string is published.	Allowed with exceptions by the catch-all group closed: /media-resize/, /admin/, /*/admin/ +96	Not probed	—

Help Reading the roster: One row per bot: the file's verdict, then the server's. Hover a bot's name for the user-agent we sent, and a verdict for its full detail. The last column says what actually stopped it. Read more

Section D · Scored group

Google's own robots

Critical

0 of 10 bots read your site.

Guide

This section answers: Can Google's other robots (images, ads, shopping) read my pages? Open guide

Words used

Gate 1
Gate 2
Captcha wall

Measured Nov. 2, 2026, 2:04 a.m.

Help Who these bots are: Google's specialist crawlers, beyond the main Googlebot: images, video, the Shopping store checker, the Search Console inspection tool and the GoogleOther family. Each one feeds a different surface, and each can be blocked separately — often by accident. Read more

1/4 scored points pass. Weakest point: Server-crawl-live-test — the live HEAD request (0%).

Your robots.txt says yes to all 10. Your server says no to 8.

10 bots knock C. Google Bots

Gate 1 Your robots.txt

10 — allowed with exceptions

7 of them have a group of their own in your file, 3 ride on the catch-all group (User-agent: *) and 0 are named by no rule. 10 of them are allowed with exceptions, and for 1 the exceptions close known content pages: /en/orders/photo-to-painting/corporate/, /en/orders/photo-to-painting/family/, /en/orders/photo-to-painting/startup/, /en/orders/photo_to_restyled_masterpiece/.

Gate 2 Your server, live

5 3 2

Captcha wallRefusedNot probed

8 Turned away by the server Gemini-Deep-Research HTTP 403Google-InspectionTool HTTP 403Storebot-Google HTTP 403GoogleOther CaptchaGoogleOther-Image CaptchaGoogleOther-Video CaptchaGooglebot-Image CaptchaGooglebot-Video Captcha

2 Judged on the file alone Googlebot-DiscoveryGooglebot-News

Their operator publishes no user-agent string, so no honest knock is possible.

Help Reading the doors: Gate 1 is your robots.txt, gate 2 your server's live answer, then the outcome each bot got. Hatched red = a captcha wall; grey = not measurable. Read more

Help Why this group is scored: Shut out here, you lose traffic, citations or link previews — so a refusal is painted red and costs points. Read more

What the measurement means, point by point

Good Server-robots-instructions — what robots.txt permits 100%

CheckedThe first thing every well-behaved bot reads. A Disallow rule on the homepage stops it before it ever requests a page, so nothing further down can compensate. A group that closes some paths but not the homepage is 'allowed with exceptions' — it counts as allowed, and the exceptions are tested against your known content pages below. A bot no rule names is allowed — silence is permission — and shown as 'not indicated'.

Foundall 10 are allowed by robots.txt; 10 of them are allowed with exceptions, and for 1 the exceptions close known content pages: /en/orders/photo-to-painting/corporate/, /en/orders/photo-to-painting/family/, /en/orders/photo-to-painting/startup/, /en/orders/photo_to_restyled_masterpiece/.

Fix soon Exceptions that close content pages 70%

CheckedWhether the paths a robots.txt group closes are housekeeping (admin, cart, search results) or the pages you want found. Each bot's exceptions are tested against this site's known content pages; a closed content page is invisible to that bot whatever the homepage verdict says.

Found1 bot(s) have exceptions closing 4 of the 500 known content pages — Googlebot-Image — on /en/orders/photo-to-painting/corporate/, /en/orders/photo-to-painting/family/, /en/orders/photo-to-painting/startup/, /en/orders/photo_to_restyled_masterpiece/

Do thisOpen robots.txt and look at the User-agent group named for each of these bots in the table below: one of its Disallow lines matches a page you publish (/en/orders/photo-to-painting/corporate/, /en/orders/photo-to-painting/family/, /en/orders/photo-to-painting/startup/ and 1 more). Narrow the pattern to the private path you meant, or add an Allow line for the content section — unless hiding those pages from these bots is deliberate.

Critical Server-crawl-live-test — the live HEAD request 0%

CheckedWhat your server really answers to one HEAD request wearing that bot's name. A firewall, a CDN rule or a bot-protection product can refuse a crawler your robots.txt welcomes — and robots.txt alone can never reveal it.

Found8 forbidden: Gemini-Deep-Research, Google-InspectionTool, GoogleOther, GoogleOther-Image, GoogleOther-Video, Googlebot-Image, Googlebot-Video, Storebot-Google

Do thisRead this one before acting: an ordinary desktop browser was refused too, so your server is not singling out crawlers — it is refusing our test requests wholesale (a rate limit, a geo or IP rule, or a challenge page). Check what your firewall does with unfamiliar visitors before changing any bot rule.

Worth knowing Do the two tests agree?

CheckedWhether the server-robots-instructions and the server-crawl-live-test tell the same story. They often do not: robots.txt is a published promise, the live answer is what actually happens, and only the comparison shows which one to believe.

Found8 bot(s) invited by robots.txt but refused by the server — the same ones the live test above names

Worth knowingA contradiction is worth resolving whichever way you meant it. Invited-but-refused: your firewall overrules your robots.txt — align them, or accept that the file is advertising a welcome the server does not honour. Blocked-but-served: your robots.txt line is a polite request only; a bot that ignores it gets the page, so add a server rule if the block really matters.

Critical The bots that decide the outcome 0%

CheckedNot every bot in this group is worth the same. These are the ones whose verdict changes what your business gets: the indexes people actually search and the assistants they actually ask.

FoundGoogle-InspectionTool is stopped by the server (HTTP 403); GoogleOther is stopped by a bot wall instead of the page — redirected to a captcha page; Googlebot-Image is stopped by a bot wall instead of the page — redirected to a captcha page

Do thisFix these before anything else in this report: each one is a whole audience that cannot see the site today. The table below names the exact rule or answer responsible for each.

Worth knowing Judged on robots.txt alone

CheckedSome operators publish a robots.txt token but never a literal user-agent string. Those bots cannot be impersonated honestly, so they are judged on your rules only — no live request was made in their name.

Found2 of 10: Googlebot-Discovery, Googlebot-News

Worth knowingNothing to do. Their robots.txt verdict is still exact — it is read from your own file — only the server-side half is unknown.

Help Reading a point: Each point says what was checked, what was found on your site, and what to do. Faults come first; Info points are statements, not passes or fails. Read more

The C. Google Bots group, bot by bot

✗

Gemini-Deep-Research

robots.txt: Allowed with exceptions · your server: HTTP 403 · the server (HTTP 403)
✗

Google-InspectionTool

robots.txt: Allowed with exceptions · your server: HTTP 403 · the server (HTTP 403)
✗

GoogleOther

robots.txt: Allowed with exceptions · your server: Captcha · a bot wall instead of the page — redirected to a captcha page
✗

GoogleOther-Image

robots.txt: Allowed with exceptions · your server: Captcha · a bot wall instead of the page — redirected to a captcha page
✗

GoogleOther-Video

robots.txt: Allowed with exceptions · your server: Captcha · a bot wall instead of the page — redirected to a captcha page
!

Googlebot-Discovery

robots.txt: Allowed with exceptions · your server: Not probed
✗

Googlebot-Image

robots.txt: Allowed with exceptions · your server: Captcha · a bot wall instead of the page — redirected to a captcha page
!

Googlebot-News

robots.txt: Allowed with exceptions · your server: Not probed
✗

Googlebot-Video

robots.txt: Allowed with exceptions · your server: Captcha · a bot wall instead of the page — redirected to a captcha page
✗

Storebot-Google

robots.txt: Allowed with exceptions · your server: HTTP 403 · the server (HTTP 403)

See the roster table

Every bot of the C. Google Bots group, with its two verdicts
Bot	server-robots-instructions	server-crawl-live-test	What stopped it
Gemini-Deep-Research	Allowed with exceptions by the catch-all group closed: /media-resize/, /admin/, /*/admin/ +96	HTTP 403	the server (HTTP 403)
Google-InspectionTool What Search Console's URL Inspection uses. A mobile variant wraps it in a Nexus 5X Chrome string.	Allowed with exceptions by the catch-all group closed: /media-resize/, /admin/, /*/admin/ +96	HTTP 403	the server (HTTP 403)
GoogleOther A bare token; may also appear via standard Googlebot-style strings.	Allowed with exceptions by `User-agent: googleother` closed: /admin/, /*/admin/, /manage/ +95	Captcha	a bot wall instead of the page — redirected to a captcha page
GoogleOther-Image	Allowed with exceptions by `User-agent: googleother` closed: /admin/, /*/admin/, /manage/ +95	Captcha	a bot wall instead of the page — redirected to a captcha page
GoogleOther-Video	Allowed with exceptions by `User-agent: googleother` closed: /admin/, /*/admin/, /manage/ +95	Captcha	a bot wall instead of the page — redirected to a captcha page
Googlebot-Discovery alias — same bot as another row No documentation found for this exact name. Discover surfacing uses the standard Googlebot / Googlebot-Image, so this is likely a naming confusion.	Allowed with exceptions by `User-agent: googlebot` closed: /admin/, /*/admin/, /manage/ +95	Not probed	—
Googlebot-Image	Allowed with exceptions by `User-agent: googlebot-image` closed: /admin/, /*/admin/, /manage/ +9	Captcha	a bot wall instead of the page — redirected to a captcha page
Googlebot-News alias — same bot as another row A distinct robots.txt token with no separate string: the requests use the standard Googlebot user-agent.	Allowed with exceptions by `User-agent: googlebot` closed: /admin/, /*/admin/, /manage/ +95	Not probed	—
Googlebot-Video	Allowed with exceptions by `User-agent: googlebot` closed: /admin/, /*/admin/, /manage/ +95	Captcha	a bot wall instead of the page — redirected to a captcha page
Storebot-Google An Android/mobile variant also exists.	Allowed with exceptions by the catch-all group closed: /media-resize/, /admin/, /*/admin/ +96	HTTP 403	the server (HTTP 403)

Help Reading the roster: One row per bot: the file's verdict, then the server's. Hover a bot's name for the user-agent we sent, and a verdict for its full detail. The last column says what actually stopped it. Read more

Section E

Other robots

Good

Guide

This section answers: What about the other robots? Open guide

Social networks, SEO tools, scrapers and cloud services: worth knowing, rarely anything to do.

Social Bots

Good

7 of 11 bots read your site.

Guide

This section answers: Which bots of this group can read your site, and what stops the others? Open guide

Words used

Gate 1
Gate 2
Captcha wall

Measured Nov. 2, 2026, 2:04 a.m.

Help Who these bots are: The fetchers that build the link preview when someone shares your page — the title, the description and the image that appear in a Facebook post, a tweet, a LinkedIn share or a Slack message. Block them and your links are shared as bare grey URLs, which people click far less. Read more

2/4 scored points pass. Weakest point: Server-crawl-live-test — the live HEAD request (64%).

Your robots.txt says yes to all 11. Your server says no to 4.

11 bots knock D. Social Bots

Gate 1 Your robots.txt

11 — allowed with exceptions

5 of them have a group of their own in your file, 6 ride on the catch-all group (User-agent: *) and 0 are named by no rule. 11 of them are allowed with exceptions, and for 2 the exceptions close known content pages: /en/orders/photo-to-painting/corporate/, /en/orders/photo-to-painting/family/, /en/orders/photo-to-painting/startup/, /en/orders/photo_to_restyled_masterpiece/.

Gate 2 Your server, live

7 4

ServedRefused

7 Read your site FacebookBotLinkedInBotMeta-ExternalAgentPinterestbotSlackbotTwitterbotfacebookexternalhit

4 Turned away by the server Meta-ExternalAds HTTP 403Meta-ExternalFetcher HTTP 403Meta-WebIndexer HTTP 403Quora-Bot HTTP 403

Help Reading the doors: Gate 1 is your robots.txt, gate 2 your server's live answer, then the outcome each bot got. Hatched red = a captcha wall; grey = not measurable. Read more

Help Why this group is scored: Shut out here, you lose traffic, citations or link previews — so a refusal is painted red and costs points. Read more

What the measurement means, point by point

Good Server-robots-instructions — what robots.txt permits 100%

CheckedThe first thing every well-behaved bot reads. A Disallow rule on the homepage stops it before it ever requests a page, so nothing further down can compensate. A group that closes some paths but not the homepage is 'allowed with exceptions' — it counts as allowed, and the exceptions are tested against your known content pages below. A bot no rule names is allowed — silence is permission — and shown as 'not indicated'.

Foundall 11 are allowed by robots.txt; 11 of them are allowed with exceptions, and for 2 the exceptions close known content pages: /en/orders/photo-to-painting/corporate/, /en/orders/photo-to-painting/family/, /en/orders/photo-to-painting/startup/, /en/orders/photo_to_restyled_masterpiece/.

Fix soon Exceptions that close content pages 70%

CheckedWhether the paths a robots.txt group closes are housekeeping (admin, cart, search results) or the pages you want found. Each bot's exceptions are tested against this site's known content pages; a closed content page is invisible to that bot whatever the homepage verdict says.

Found2 bot(s) have exceptions closing 4 of the 500 known content pages — Twitterbot, facebookexternalhit — on /en/orders/photo-to-painting/corporate/, /en/orders/photo-to-painting/family/, /en/orders/photo-to-painting/startup/, /en/orders/photo_to_restyled_masterpiece/

Do thisOpen robots.txt and look at the User-agent group named for each of these bots in the table below: one of its Disallow lines matches a page you publish (/en/orders/photo-to-painting/corporate/, /en/orders/photo-to-painting/family/, /en/orders/photo-to-painting/startup/ and 1 more). Narrow the pattern to the private path you meant, or add an Allow line for the content section — unless hiding those pages from these bots is deliberate.

Fix soon Server-crawl-live-test — the live HEAD request 64%

CheckedWhat your server really answers to one HEAD request wearing that bot's name. A firewall, a CDN rule or a bot-protection product can refuse a crawler your robots.txt welcomes — and robots.txt alone can never reveal it.

Found4 forbidden: Meta-ExternalAds, Meta-ExternalFetcher, Meta-WebIndexer, Quora-Bot

Do thisRead this one before acting: an ordinary desktop browser was refused too, so your server is not singling out crawlers — it is refusing our test requests wholesale (a rate limit, a geo or IP rule, or a challenge page). Check what your firewall does with unfamiliar visitors before changing any bot rule.

Worth knowing Indexing directives in the headers

CheckedThe X-Robots-Tag response header of the answers your server actually served. A noindex here lets the bot in and then tells it to throw the page away — the most expensive way to be invisible, because it looks fine in every other check. Only served answers are read: a wall's or an error page's header is not yours.

Foundno X-Robots-Tag header in any served answer

Worth knowing Do the two tests agree?

CheckedWhether the server-robots-instructions and the server-crawl-live-test tell the same story. They often do not: robots.txt is a published promise, the live answer is what actually happens, and only the comparison shows which one to believe.

Found4 bot(s) invited by robots.txt but refused by the server — the same ones the live test above names

Worth knowingA contradiction is worth resolving whichever way you meant it. Invited-but-refused: your firewall overrules your robots.txt — align them, or accept that the file is advertising a welcome the server does not honour. Blocked-but-served: your robots.txt line is a polite request only; a bot that ignores it gets the page, so add a server rule if the block really matters.

Good The bots that decide the outcome 100%

CheckedNot every bot in this group is worth the same. These are the ones whose verdict changes what your business gets: the indexes people actually search and the assistants they actually ask.

Foundnone of the 5 is blocked or refused

Help Reading a point: Each point says what was checked, what was found on your site, and what to do. Faults come first; Info points are statements, not passes or fails. Read more

The D. Social Bots group, bot by bot

✓

FacebookBot

robots.txt: Allowed with exceptions · your server: Redirected
✓

LinkedInBot

robots.txt: Allowed with exceptions · your server: Redirected
✗

Meta-ExternalAds

robots.txt: Allowed with exceptions · your server: HTTP 403 · the server (HTTP 403)
✓

Meta-ExternalAgent

robots.txt: Allowed with exceptions · your server: Redirected
✗

Meta-ExternalFetcher

robots.txt: Allowed with exceptions · your server: HTTP 403 · the server (HTTP 403)
✗

Meta-WebIndexer

robots.txt: Allowed with exceptions · your server: HTTP 403 · the server (HTTP 403)
✓

Pinterestbot

robots.txt: Allowed with exceptions · your server: Redirected
✗

Quora-Bot

robots.txt: Allowed with exceptions · your server: HTTP 403 · the server (HTTP 403)
✓

Slackbot

robots.txt: Allowed with exceptions · your server: Redirected
✓

Twitterbot

robots.txt: Allowed with exceptions · your server: Redirected
✓

facebookexternalhit

robots.txt: Allowed with exceptions · your server: Redirected

See the roster table

Every bot of the D. Social Bots group, with its two verdicts
Bot	server-robots-instructions	server-crawl-live-test	What stopped it
FacebookBot Meta's general research/product crawler, distinct from facebookexternalhit.	Allowed with exceptions by `User-agent: facebookbot` closed: /admin/, /*/admin/, /manage/ +95	Redirected	—
LinkedInBot The HttpClient version field varies.	Allowed with exceptions by `User-agent: linkedinbot` closed: /admin/, /*/admin/, /manage/ +95	Redirected	—
Meta-ExternalAds Ad-landing-page review.	Allowed with exceptions by the catch-all group closed: /media-resize/, /admin/, /*/admin/ +96	HTTP 403	the server (HTTP 403)
Meta-ExternalAgent AI training plus search indexing.	Allowed with exceptions by `User-agent: meta-externalagent` closed: /admin/, /*/admin/, /manage/ +95	Redirected	—
Meta-ExternalFetcher On-demand fetch for Meta AI features.	Allowed with exceptions by the catch-all group closed: /media-resize/, /admin/, /*/admin/ +96	HTTP 403	the server (HTTP 403)
Meta-WebIndexer Meta's newer (2025) general web-indexing crawler.	Allowed with exceptions by the catch-all group closed: /media-resize/, /admin/, /*/admin/ +96	HTTP 403	the server (HTTP 403)
Pinterestbot A lighter fetcher is also seen: Pinterest/0.2 (+https://www.pinterest.com/bot.html).	Allowed with exceptions by the catch-all group closed: /media-resize/, /admin/, /*/admin/ +96	Redirected	—
Quora-Bot	Allowed with exceptions by the catch-all group closed: /media-resize/, /admin/, /*/admin/ +96	HTTP 403	the server (HTTP 403)
Slackbot Other Slack integrations use Slackbot 1.0 (+https://api.slack.com/robots).	Allowed with exceptions by the catch-all group closed: /media-resize/, /admin/, /*/admin/ +96	Redirected	—
Twitterbot	Allowed with exceptions by `User-agent: twitterbot` closed: /admin/, /*/admin/, /manage/ +9	Redirected	its exceptions close 4 content pages: /en/orders/photo-to-painting/corporate/, /en/orders/photo-to-painting/family/, /en/orders/photo-to-painting/startup/, /en/orders/photo_to_restyled_masterpiece/
facebookexternalhit The link-preview fetcher; also seen bare as facebookexternalhit/1.1.	Allowed with exceptions by `User-agent: facebookexternalhit` closed: /admin/, /*/admin/, /manage/ +9	Redirected	its exceptions close 4 content pages: /en/orders/photo-to-painting/corporate/, /en/orders/photo-to-painting/family/, /en/orders/photo-to-painting/startup/, /en/orders/photo_to_restyled_masterpiece/

Help Reading the roster: One row per bot: the file's verdict, then the server's. Hover a bot's name for the user-agent we sent, and a verdict for its full detail. The last column says what actually stopped it. Read more

SEO Tools

0 of 21 bots read your site.

Guide

This section answers: Which bots of this group can read your site, and what stops the others? Open guide

Words used

Gate 1
Gate 2
Captcha wall

Measured Nov. 2, 2026, 2:04 a.m.

Help Who these bots are: Backlink and audit crawlers: Ahrefs, Semrush, Majestic, Moz and the rest. Informational, not scored — blocking them saves bandwidth and hides your link profile from competitors, allowing them lets those tools (and the reports built on them, including some of AkOOre's data) see your site. Both are defensible. Read more

Your robots.txt says yes to all 21. Your server says no to 20.

21 bots knock E. SEO Tools

Gate 1 Your robots.txt

21 — allowed with exceptions

All of them ride on your catch-all group (User-agent: *) — no rule in your file names one of them individually. 21 of them are allowed with exceptions — the closed paths are housekeeping only, no known content page is among them.

Gate 2 Your server, live

20 1

RefusedNot probed

20 Turned away by the server AhrefsBot HTTP 403AudigentAdBot HTTP 403AwarioRssBot HTTP 403AwarioSmartBot HTTP 403Brandwatch HTTP 403Chrome-Lighthouse HTTP 403DataForSeoBot HTTP 403DotBot HTTP 403Google Page Speed Insights HTTP 403Jetslide HTTP 403 + 10 more

1 Judged on the file alone NeticleBot

Their operator publishes no user-agent string, so no honest knock is possible.

Help Reading the doors: Gate 1 is your robots.txt, gate 2 your server's live answer, then the outcome each bot got. Hatched red = a captcha wall; grey = not measurable. Read more

Help Why this group is not scored: Blocking these bots is a legitimate decision about your own content: a shut door is drawn in ink, never marked wrong. Read more

What the measurement means, point by point

Worth knowing Server-robots-instructions — what robots.txt permits

CheckedThe first thing every well-behaved bot reads. A Disallow rule on the homepage stops it before it ever requests a page, so nothing further down can compensate. A group that closes some paths but not the homepage is 'allowed with exceptions' — it counts as allowed, and the exceptions are tested against your known content pages below. A bot no rule names is allowed — silence is permission — and shown as 'not indicated'.

Foundall 21 are allowed by robots.txt; 21 of them are allowed with exceptions — the closed paths are housekeeping only, no known content page is among them.

Worth knowing Server-crawl-live-test — the live HEAD request

CheckedWhat your server really answers to one HEAD request wearing that bot's name. A firewall, a CDN rule or a bot-protection product can refuse a crawler your robots.txt welcomes — and robots.txt alone can never reveal it.

Found20 forbidden: AhrefsBot, AudigentAdBot, AwarioRssBot, AwarioSmartBot, Brandwatch, Chrome-Lighthouse, DataForSeoBot, DotBot and 12 more

Worth knowingRead this one before acting: an ordinary desktop browser was refused too, so your server is not singling out crawlers — it is refusing our test requests wholesale (a rate limit, a geo or IP rule, or a challenge page). Check what your firewall does with unfamiliar visitors before changing any bot rule.

Worth knowing Do the two tests agree?

CheckedWhether the server-robots-instructions and the server-crawl-live-test tell the same story. They often do not: robots.txt is a published promise, the live answer is what actually happens, and only the comparison shows which one to believe.

Found20 bot(s) invited by robots.txt but refused by the server — the same ones the live test above names

Worth knowingA contradiction is worth resolving whichever way you meant it. Invited-but-refused: your firewall overrules your robots.txt — align them, or accept that the file is advertising a welcome the server does not honour. Blocked-but-served: your robots.txt line is a polite request only; a bot that ignores it gets the page, so add a server rule if the block really matters.

Worth knowing Judged on robots.txt alone

CheckedSome operators publish a robots.txt token but never a literal user-agent string. Those bots cannot be impersonated honestly, so they are judged on your rules only — no live request was made in their name.

Found1 of 21: NeticleBot

Worth knowingNothing to do. Their robots.txt verdict is still exact — it is read from your own file — only the server-side half is unknown.

Help Reading a point: Each point says what was checked, what was found on your site, and what to do. Faults come first; Info points are statements, not passes or fails. Read more

The E. SEO Tools group, bot by bot

✗

AhrefsBot

robots.txt: Allowed with exceptions · your server: HTTP 403 · the server (HTTP 403)
✗

AudigentAdBot

robots.txt: Allowed with exceptions · your server: HTTP 403 · the server (HTTP 403)
✗

AwarioRssBot

robots.txt: Allowed with exceptions · your server: HTTP 403 · the server (HTTP 403)
✗

AwarioSmartBot

robots.txt: Allowed with exceptions · your server: HTTP 403 · the server (HTTP 403)
✗

Brandwatch

robots.txt: Allowed with exceptions · your server: HTTP 403 · the server (HTTP 403)
✗

Chrome-Lighthouse

robots.txt: Allowed with exceptions · your server: HTTP 403 · the server (HTTP 403)
✗

DataForSeoBot

robots.txt: Allowed with exceptions · your server: HTTP 403 · the server (HTTP 403)
✗

DotBot

robots.txt: Allowed with exceptions · your server: HTTP 403 · the server (HTTP 403)
✗

Google Page Speed Insights

robots.txt: Allowed with exceptions · your server: HTTP 403 · the server (HTTP 403)
✗

Jetslide

robots.txt: Allowed with exceptions · your server: HTTP 403 · the server (HTTP 403)
✗

MJ12bot

robots.txt: Allowed with exceptions · your server: HTTP 403 · the server (HTTP 403)
✗

Meltwater

robots.txt: Allowed with exceptions · your server: HTTP 403 · the server (HTTP 403)
!

NeticleBot

robots.txt: Allowed with exceptions · your server: Not probed
✗

Netvibes

robots.txt: Allowed with exceptions · your server: HTTP 403 · the server (HTTP 403)
✗

Screaming Frog SEO Spider

robots.txt: Allowed with exceptions · your server: HTTP 403 · the server (HTTP 403)
✗

SearchmetricsBot

robots.txt: Allowed with exceptions · your server: HTTP 403 · the server (HTTP 403)
✗

SemrushBot

robots.txt: Allowed with exceptions · your server: HTTP 403 · the server (HTTP 403)
✗

SemrushBot-OCOB

robots.txt: Allowed with exceptions · your server: HTTP 403 · the server (HTTP 403)
✗

SemrushBotSwa

robots.txt: Allowed with exceptions · your server: HTTP 403 · the server (HTTP 403)
✗

Sidetrade Indexer Bot

robots.txt: Allowed with exceptions · your server: HTTP 403 · the server (HTTP 403)
✗

peer39_crawler

robots.txt: Allowed with exceptions · your server: HTTP 403 · the server (HTTP 403)

See the roster table

Every bot of the E. SEO Tools group, with its two verdicts
Bot	server-robots-instructions	server-crawl-live-test	What stopped it
AhrefsBot	Allowed with exceptions by the catch-all group closed: /media-resize/, /admin/, /*/admin/ +96	HTTP 403	the server (HTTP 403)
AudigentAdBot	Allowed with exceptions by the catch-all group closed: /media-resize/, /admin/, /*/admin/ +96	HTTP 403	the server (HTTP 403)
AwarioRssBot	Allowed with exceptions by the catch-all group closed: /media-resize/, /admin/, /*/admin/ +96	HTTP 403	the server (HTTP 403)
AwarioSmartBot	Allowed with exceptions by the catch-all group closed: /media-resize/, /admin/, /*/admin/ +96	HTTP 403	the server (HTTP 403)
Brandwatch The vendor's own page says the string varies, so this literal is low confidence.	Allowed with exceptions by the catch-all group closed: /media-resize/, /admin/, /*/admin/ +96	HTTP 403	the server (HTTP 403)
Chrome-Lighthouse	Allowed with exceptions by the catch-all group closed: /media-resize/, /admin/, /*/admin/ +96	HTTP 403	the server (HTTP 403)
DataForSeoBot The crawler behind the data AkOOre's own reports are built on.	Allowed with exceptions by the catch-all group closed: /media-resize/, /admin/, /*/admin/ +96	HTTP 403	the server (HTTP 403)
DotBot A newer DotBot/2.0; +https://dotbot.com/about/ form is also reported but unconfirmed.	Allowed with exceptions by the catch-all group closed: /media-resize/, /admin/, /*/admin/ +96	HTTP 403	the server (HTTP 403)
Google Page Speed Insights alias — same bot as another row The documented legacy form. Recent PSI requests reportedly drop these tokens for a plain Chrome user-agent.	Allowed with exceptions by the catch-all group closed: /media-resize/, /admin/, /*/admin/ +96	HTTP 403	the server (HTTP 403)
Jetslide	Allowed with exceptions by the catch-all group closed: /media-resize/, /admin/, /*/admin/ +96	HTTP 403	the server (HTTP 403)
MJ12bot Newer v2.0.x builds are also reported, same format.	Allowed with exceptions by the catch-all group closed: /media-resize/, /admin/, /*/admin/ +96	HTTP 403	the server (HTTP 403)
Meltwater	Allowed with exceptions by the catch-all group closed: /media-resize/, /admin/, /*/admin/ +96	HTTP 403	the server (HTTP 403)
NeticleBot The vendor has a bot-info page but quotes no exact header.	Allowed with exceptions by the catch-all group closed: /media-resize/, /admin/, /*/admin/ +96	Not probed	—
Netvibes	Allowed with exceptions by the catch-all group closed: /media-resize/, /admin/, /*/admin/ +96	HTTP 403	the server (HTTP 403)
Screaming Frog SEO Spider The version tracks the installed app and changes every release.	Allowed with exceptions by the catch-all group closed: /media-resize/, /admin/, /*/admin/ +96	HTTP 403	the server (HTTP 403)
SearchmetricsBot	Allowed with exceptions by the catch-all group closed: /media-resize/, /admin/, /*/admin/ +96	HTTP 403	the server (HTTP 403)
SemrushBot Older SemrushBot-BA / -SA / -SI variants also exist.	Allowed with exceptions by the catch-all group closed: /media-resize/, /admin/, /*/admin/ +96	HTTP 403	the server (HTTP 403)
SemrushBot-OCOB The Content Toolkit crawler.	Allowed with exceptions by the catch-all group closed: /media-resize/, /admin/, /*/admin/ +96	HTTP 403	the server (HTTP 403)
SemrushBotSwa	Allowed with exceptions by the catch-all group closed: /media-resize/, /admin/, /*/admin/ +96	HTTP 403	the server (HTTP 403)
Sidetrade Indexer Bot	Allowed with exceptions by the catch-all group closed: /media-resize/, /admin/, /*/admin/ +96	HTTP 403	the server (HTTP 403)
peer39_crawler Ad-context classification. The source list names it twice (SEO Tools and Other Agents); it is one bot, listed once.	Allowed with exceptions by the catch-all group closed: /media-resize/, /admin/, /*/admin/ +96	HTTP 403	the server (HTTP 403)

Help Reading the roster: One row per bot: the file's verdict, then the server's. Hover a bot's name for the user-agent we sent, and a verdict for its full detail. The last column says what actually stopped it. Read more

Scrapers

1 of 46 bots read your site.

Guide

This section answers: Which bots of this group can read your site, and what stops the others? Open guide

Words used

Gate 1
Gate 2
Captcha wall

Measured Nov. 2, 2026, 2:04 a.m.

Help Who these bots are: Content harvesters, archivers, research crawlers and dataset builders. Informational, not scored: blocking them is a legitimate decision about what happens to your content, and this section only shows you what your current rules actually do. Read more

Your robots.txt says yes to all 46. Your server says no to 34.

46 bots knock F. Scrapers

Gate 1 Your robots.txt

46 — allowed with exceptions

All of them ride on your catch-all group (User-agent: *) — no rule in your file names one of them individually. 46 of them are allowed with exceptions — the closed paths are housekeeping only, no known content page is among them.

Gate 2 Your server, live

1 1 33 11

ServedCaptcha wallRefusedNot probed

1 Read your site Barkrowler

34 Turned away by the server Arquivo-web-crawler HTTP 403BLEXBot HTTP 403Bytespider HTTP 403Cotoyogi HTTP 403Crawl4AI HTTP 403Diffbot HTTP 403EchoboxBot HTTP 403Factset_spyderbot HTTP 403Firecrawl HTTP 403FriendlyCrawler HTTP 403 + 24 more

11 Judged on the file alone Amazon KendraBravestCrawlspaceEchobot BotJenkersBotPoseidon Research CrawlerSeekrSeekrBothada.newsimediaethics.org + 1 more

Their operator publishes no user-agent string, so no honest knock is possible.

Help Reading the doors: Gate 1 is your robots.txt, gate 2 your server's live answer, then the outcome each bot got. Hatched red = a captcha wall; grey = not measurable. Read more

Help Why this group is not scored: Blocking these bots is a legitimate decision about your own content: a shut door is drawn in ink, never marked wrong. Read more

What the measurement means, point by point

Worth knowing Server-robots-instructions — what robots.txt permits

CheckedThe first thing every well-behaved bot reads. A Disallow rule on the homepage stops it before it ever requests a page, so nothing further down can compensate. A group that closes some paths but not the homepage is 'allowed with exceptions' — it counts as allowed, and the exceptions are tested against your known content pages below. A bot no rule names is allowed — silence is permission — and shown as 'not indicated'.

Foundall 46 are allowed by robots.txt; 46 of them are allowed with exceptions — the closed paths are housekeeping only, no known content page is among them.

Worth knowing Server-crawl-live-test — the live HEAD request

CheckedWhat your server really answers to one HEAD request wearing that bot's name. A firewall, a CDN rule or a bot-protection product can refuse a crawler your robots.txt welcomes — and robots.txt alone can never reveal it.

Found34 forbidden: Arquivo-web-crawler, BLEXBot, Bytespider, CCBot, Cotoyogi, Crawl4AI, Diffbot, EchoboxBot and 26 more

Worth knowingRead this one before acting: an ordinary desktop browser was refused too, so your server is not singling out crawlers — it is refusing our test requests wholesale (a rate limit, a geo or IP rule, or a challenge page). Check what your firewall does with unfamiliar visitors before changing any bot rule.

Worth knowing Indexing directives in the headers

CheckedThe X-Robots-Tag response header of the answers your server actually served. A noindex here lets the bot in and then tells it to throw the page away — the most expensive way to be invisible, because it looks fine in every other check. Only served answers are read: a wall's or an error page's header is not yours.

Foundno X-Robots-Tag header in any served answer

Worth knowing Do the two tests agree?

CheckedWhether the server-robots-instructions and the server-crawl-live-test tell the same story. They often do not: robots.txt is a published promise, the live answer is what actually happens, and only the comparison shows which one to believe.

Found34 bot(s) invited by robots.txt but refused by the server — the same ones the live test above names

Worth knowingA contradiction is worth resolving whichever way you meant it. Invited-but-refused: your firewall overrules your robots.txt — align them, or accept that the file is advertising a welcome the server does not honour. Blocked-but-served: your robots.txt line is a polite request only; a bot that ignores it gets the page, so add a server rule if the block really matters.

Worth knowing Judged on robots.txt alone

CheckedSome operators publish a robots.txt token but never a literal user-agent string. Those bots cannot be impersonated honestly, so they are judged on your rules only — no live request was made in their name.

Found11 of 46: Amazon Kendra, Bravest, Crawlspace, Echobot Bot, JenkersBot, Poseidon Research Crawler, Seekr, SeekrBot and 3 more

Worth knowingNothing to do. Their robots.txt verdict is still exact — it is read from your own file — only the server-side half is unknown.

Help Reading a point: Each point says what was checked, what was found on your site, and what to do. Faults come first; Info points are statements, not passes or fails. Read more

The F. Scrapers group, bot by bot

!

Amazon Kendra

robots.txt: Allowed with exceptions · your server: Not probed
✗

Arquivo-web-crawler

robots.txt: Allowed with exceptions · your server: HTTP 403 · the server (HTTP 403)
✗

BLEXBot

robots.txt: Allowed with exceptions · your server: HTTP 403 · the server (HTTP 403)
✓

Barkrowler

robots.txt: Allowed with exceptions · your server: Redirected
!

Bravest

robots.txt: Allowed with exceptions · your server: Not probed
✗

Bytespider

robots.txt: Allowed with exceptions · your server: HTTP 403 · the server (HTTP 403)
✗

CCBot

robots.txt: Allowed with exceptions · your server: Captcha · a bot wall instead of the page — redirected to a captcha page
✗

Cotoyogi

robots.txt: Allowed with exceptions · your server: HTTP 403 · the server (HTTP 403)
✗

Crawl4AI

robots.txt: Allowed with exceptions · your server: HTTP 403 · the server (HTTP 403)
!

Crawlspace

robots.txt: Allowed with exceptions · your server: Not probed
✗

Diffbot

robots.txt: Allowed with exceptions · your server: HTTP 403 · the server (HTTP 403)
!

Echobot Bot

robots.txt: Allowed with exceptions · your server: Not probed
✗

EchoboxBot

robots.txt: Allowed with exceptions · your server: HTTP 403 · the server (HTTP 403)
✗

Factset_spyderbot

robots.txt: Allowed with exceptions · your server: HTTP 403 · the server (HTTP 403)
✗

Firecrawl

robots.txt: Allowed with exceptions · your server: HTTP 403 · the server (HTTP 403)
✗

FriendlyCrawler

robots.txt: Allowed with exceptions · your server: HTTP 403 · the server (HTTP 403)
✗

ICC-Crawler

robots.txt: Allowed with exceptions · your server: HTTP 403 · the server (HTTP 403)
✗

ISSCyberRiskCrawler

robots.txt: Allowed with exceptions · your server: HTTP 403 · the server (HTTP 403)
✗

ImagesiftBot

robots.txt: Allowed with exceptions · your server: HTTP 403 · the server (HTTP 403)
!

JenkersBot

robots.txt: Allowed with exceptions · your server: Not probed
✗

Kangaroo Bot

robots.txt: Allowed with exceptions · your server: HTTP 403 · the server (HTTP 403)
✗

NewsNow

robots.txt: Allowed with exceptions · your server: HTTP 403 · the server (HTTP 403)
✗

NovaAct

robots.txt: Allowed with exceptions · your server: HTTP 403 · the server (HTTP 403)
!

Poseidon Research Crawler

robots.txt: Allowed with exceptions · your server: Not probed
✗

QualifiedBot

robots.txt: Allowed with exceptions · your server: HTTP 403 · the server (HTTP 403)
✗

Scrapy

robots.txt: Allowed with exceptions · your server: HTTP 403 · the server (HTTP 403)
✗

SeekportBot

robots.txt: Allowed with exceptions · your server: HTTP 403 · the server (HTTP 403)
!

Seekr

robots.txt: Allowed with exceptions · your server: Not probed
!

SeekrBot

robots.txt: Allowed with exceptions · your server: Not probed
✗

TaraGroup Intelligent Bot

robots.txt: Allowed with exceptions · your server: HTTP 403 · the server (HTTP 403)
✗

Timpibot

robots.txt: Allowed with exceptions · your server: HTTP 403 · the server (HTTP 403)
✗

Turnitin

robots.txt: Allowed with exceptions · your server: HTTP 403 · the server (HTTP 403)
✗

VelenPublicWebCrawler

robots.txt: Allowed with exceptions · your server: HTTP 403 · the server (HTTP 403)
✗

Webzio-Extended

robots.txt: Allowed with exceptions · your server: HTTP 403 · the server (HTTP 403)
✗

coccocbot-web

robots.txt: Allowed with exceptions · your server: HTTP 403 · the server (HTTP 403)
✗

crawler4j

robots.txt: Allowed with exceptions · your server: HTTP 403 · the server (HTTP 403)
!

hada.news

robots.txt: Allowed with exceptions · your server: Not probed
✗

iaskspider

robots.txt: Allowed with exceptions · your server: HTTP 403 · the server (HTTP 403)
!

imediaethics.org

robots.txt: Allowed with exceptions · your server: Not probed
✗

imgproxy

robots.txt: Allowed with exceptions · your server: HTTP 403 · the server (HTTP 403)
✗

magpie-crawler

robots.txt: Allowed with exceptions · your server: HTTP 403 · the server (HTTP 403)
!

netEstate Imprint Crawler

robots.txt: Allowed with exceptions · your server: Not probed
✗

news-please

robots.txt: Allowed with exceptions · your server: HTTP 403 · the server (HTTP 403)
✗

omgili

robots.txt: Allowed with exceptions · your server: HTTP 403 · the server (HTTP 403)
✗

omgilibot

robots.txt: Allowed with exceptions · your server: HTTP 403 · the server (HTTP 403)
✗

yacybot

robots.txt: Allowed with exceptions · your server: HTTP 403 · the server (HTTP 403)

See the roster table

Every bot of the F. Scrapers group, with its two verdicts
Bot	server-robots-instructions	server-crawl-live-test	What stopped it
Amazon Kendra AWS publishes the robots.txt token but no literal HTTP header.	Allowed with exceptions by the catch-all group closed: /media-resize/, /admin/, /*/admin/ +96	Not probed	—
Arquivo-web-crawler	Allowed with exceptions by the catch-all group closed: /media-resize/, /admin/, /*/admin/ +96	HTTP 403	the server (HTTP 403)
BLEXBot	Allowed with exceptions by the catch-all group closed: /media-resize/, /admin/, /*/admin/ +96	HTTP 403	the server (HTTP 403)
Barkrowler	Allowed with exceptions by the catch-all group closed: /media-resize/, /admin/, /*/admin/ +96	Redirected	—
Bravest No published user-agent string.	Allowed with exceptions by the catch-all group closed: /media-resize/, /admin/, /*/admin/ +96	Not probed	—
Bytespider ByteDance never published an official documentation page; the Bytespider token is the stable part.	Allowed with exceptions by the catch-all group closed: /media-resize/, /admin/, /*/admin/ +96	HTTP 403	the server (HTTP 403)
CCBot Common Crawl feeds most open models' training sets.	Allowed with exceptions by the catch-all group closed: /media-resize/, /admin/, /*/admin/ +96	Captcha	a bot wall instead of the page — redirected to a captcha page
Cotoyogi	Allowed with exceptions by the catch-all group closed: /media-resize/, /admin/, /*/admin/ +96	HTTP 403	the server (HTTP 403)
Crawl4AI A library default the caller can override.	Allowed with exceptions by the catch-all group closed: /media-resize/, /admin/, /*/admin/ +96	HTTP 403	the server (HTTP 403)
Crawlspace No published user-agent string.	Allowed with exceptions by the catch-all group closed: /media-resize/, /admin/, /*/admin/ +96	Not probed	—
Diffbot Version numbers beyond 0.1 are unverified.	Allowed with exceptions by the catch-all group closed: /media-resize/, /admin/, /*/admin/ +96	HTTP 403	the server (HTTP 403)
Echobot Bot Not to be confused with EchoboxBot above: different company, different bot.	Allowed with exceptions by the catch-all group closed: /media-resize/, /admin/, /*/admin/ +96	Not probed	—
EchoboxBot The hash segment is per-deployment.	Allowed with exceptions by the catch-all group closed: /media-resize/, /admin/, /*/admin/ +96	HTTP 403	the server (HTTP 403)
Factset_spyderbot	Allowed with exceptions by the catch-all group closed: /media-resize/, /admin/, /*/admin/ +96	HTTP 403	the server (HTTP 403)
Firecrawl	Allowed with exceptions by the catch-all group closed: /media-resize/, /admin/, /*/admin/ +96	HTTP 403	the server (HTTP 403)
FriendlyCrawler	Allowed with exceptions by the catch-all group closed: /media-resize/, /admin/, /*/admin/ +96	HTTP 403	the server (HTTP 403)
ICC-Crawler	Allowed with exceptions by the catch-all group closed: /media-resize/, /admin/, /*/admin/ +96	HTTP 403	the server (HTTP 403)
ISSCyberRiskCrawler	Allowed with exceptions by the catch-all group closed: /media-resize/, /admin/, /*/admin/ +96	HTTP 403	the server (HTTP 403)
ImagesiftBot	Allowed with exceptions by the catch-all group closed: /media-resize/, /admin/, /*/admin/ +96	HTTP 403	the server (HTTP 403)
JenkersBot No published user-agent string.	Allowed with exceptions by the catch-all group closed: /media-resize/, /admin/, /*/admin/ +96	Not probed	—
Kangaroo Bot A differently spelled KangarooBot/1.0; +https://kangaroobot.com (no space) is documented elsewhere -- check which one matches your logs.	Allowed with exceptions by the catch-all group closed: /media-resize/, /admin/, /*/admin/ +96	HTTP 403	the server (HTTP 403)
NewsNow	Allowed with exceptions by the catch-all group closed: /media-resize/, /admin/, /*/admin/ +96	HTTP 403	the server (HTTP 403)
NovaAct	Allowed with exceptions by the catch-all group closed: /media-resize/, /admin/, /*/admin/ +96	HTTP 403	the server (HTTP 403)
Poseidon Research Crawler Only the bare identifier is documented.	Allowed with exceptions by the catch-all group closed: /media-resize/, /admin/, /*/admin/ +96	Not probed	—
QualifiedBot	Allowed with exceptions by the catch-all group closed: /media-resize/, /admin/, /*/admin/ +96	HTTP 403	the server (HTTP 403)
Scrapy The library default template; the version tracks the installed Scrapy release.	Allowed with exceptions by the catch-all group closed: /media-resize/, /admin/, /*/admin/ +96	HTTP 403	the server (HTTP 403)
SeekportBot	Allowed with exceptions by the catch-all group closed: /media-resize/, /admin/, /*/admin/ +96	HTTP 403	the server (HTTP 403)
Seekr alias — same bot as another row No published user-agent string.	Allowed with exceptions by the catch-all group closed: /media-resize/, /admin/, /*/admin/ +96	Not probed	—
SeekrBot No published user-agent string.	Allowed with exceptions by the catch-all group closed: /media-resize/, /admin/, /*/admin/ +96	Not probed	—
TaraGroup Intelligent Bot	Allowed with exceptions by the catch-all group closed: /media-resize/, /admin/, /*/admin/ +96	HTTP 403	the server (HTTP 403)
Timpibot	Allowed with exceptions by the catch-all group closed: /media-resize/, /admin/, /*/admin/ +96	HTTP 403	the server (HTTP 403)
Turnitin	Allowed with exceptions by the catch-all group closed: /media-resize/, /admin/, /*/admin/ +96	HTTP 403	the server (HTTP 403)
VelenPublicWebCrawler	Allowed with exceptions by the catch-all group closed: /media-resize/, /admin/, /*/admin/ +96	HTTP 403	the server (HTTP 403)
Webzio-Extended	Allowed with exceptions by the catch-all group closed: /media-resize/, /admin/, /*/admin/ +96	HTTP 403	the server (HTTP 403)
coccocbot-web	Allowed with exceptions by the catch-all group closed: /media-resize/, /admin/, /*/admin/ +96	HTTP 403	the server (HTTP 403)
crawler4j A library default, set per deployment by the developer.	Allowed with exceptions by the catch-all group closed: /media-resize/, /admin/, /*/admin/ +96	HTTP 403	the server (HTTP 403)
hada.news No published user-agent string.	Allowed with exceptions by the catch-all group closed: /media-resize/, /admin/, /*/admin/ +96	Not probed	—
iaskspider	Allowed with exceptions by the catch-all group closed: /media-resize/, /admin/, /*/admin/ +96	HTTP 403	the server (HTTP 403)
imediaethics.org No published user-agent string.	Allowed with exceptions by the catch-all group closed: /media-resize/, /admin/, /*/admin/ +96	Not probed	—
imgproxy A self-hosted image-proxy tool rather than a crawler; the version tracks its release.	Allowed with exceptions by the catch-all group closed: /media-resize/, /admin/, /*/admin/ +96	HTTP 403	the server (HTTP 403)
magpie-crawler A legacy crawler name, distinct from the newer Brandwatch user-agent in SEO Tools.	Allowed with exceptions by the catch-all group closed: /media-resize/, /admin/, /*/admin/ +96	HTTP 403	the server (HTTP 403)
netEstate Imprint Crawler Not to be confused with the separately documented netEstate NE Crawler (+http://www.website-datenbank.de/), an older bot.	Allowed with exceptions by the catch-all group closed: /media-resize/, /admin/, /*/admin/ +96	Not probed	—
news-please A library default template, not a fixed signature.	Allowed with exceptions by the catch-all group closed: /media-resize/, /admin/, /*/admin/ +96	HTTP 403	the server (HTTP 403)
omgili	Allowed with exceptions by the catch-all group closed: /media-resize/, /admin/, /*/admin/ +96	HTTP 403	the server (HTTP 403)
omgilibot An older omgilibot/0.3 +http://www.omgili.com/Crawler.html form is also documented.	Allowed with exceptions by the catch-all group closed: /media-resize/, /admin/, /*/admin/ +96	HTTP 403	the server (HTTP 403)
yacybot The bracketed system info varies per peer; yacybot is the stable token.	Allowed with exceptions by the catch-all group closed: /media-resize/, /admin/, /*/admin/ +96	HTTP 403	the server (HTTP 403)

Help Reading the roster: One row per bot: the file's verdict, then the server's. Hover a bot's name for the user-agent we sent, and a verdict for its full detail. The last column says what actually stopped it. Read more

Cloud Services

0 of 4 bots read your site.

Guide

This section answers: Which bots of this group can read your site, and what stops the others? Open guide

Words used

Gate 1
Gate 2
Captcha wall

Measured Nov. 2, 2026, 2:04 a.m.

Help Who these bots are: Enterprise search and cloud indexing bots — Google Vertex, Amazonbot, Alibaba. They matter when your customers search inside a platform rather than on the open web. Informational. Read more

Your robots.txt says yes to all 4. Your server says no to 4.

4 bots knock G. Cloud Services

Gate 1 Your robots.txt

4 — allowed with exceptions

All of them ride on your catch-all group (User-agent: *) — no rule in your file names one of them individually. 4 of them are allowed with exceptions — the closed paths are housekeeping only, no known content page is among them.

Gate 2 Your server, live

4 — refused

4 Turned away by the server AliyunSecBot HTTP 403Amazonbot HTTP 403Amzn-User HTTP 403Google-CloudVertexBot HTTP 403

Help Reading the doors: Gate 1 is your robots.txt, gate 2 your server's live answer, then the outcome each bot got. Hatched red = a captcha wall; grey = not measurable. Read more

Help Why this group is not scored: Blocking these bots is a legitimate decision about your own content: a shut door is drawn in ink, never marked wrong. Read more

What the measurement means, point by point

Worth knowing Server-robots-instructions — what robots.txt permits

CheckedThe first thing every well-behaved bot reads. A Disallow rule on the homepage stops it before it ever requests a page, so nothing further down can compensate. A group that closes some paths but not the homepage is 'allowed with exceptions' — it counts as allowed, and the exceptions are tested against your known content pages below. A bot no rule names is allowed — silence is permission — and shown as 'not indicated'.

Foundall 4 are allowed by robots.txt; 4 of them are allowed with exceptions — the closed paths are housekeeping only, no known content page is among them.

Worth knowing Server-crawl-live-test — the live HEAD request

CheckedWhat your server really answers to one HEAD request wearing that bot's name. A firewall, a CDN rule or a bot-protection product can refuse a crawler your robots.txt welcomes — and robots.txt alone can never reveal it.

Found4 forbidden: AliyunSecBot, Amazonbot, Amzn-User, Google-CloudVertexBot

Worth knowingRead this one before acting: an ordinary desktop browser was refused too, so your server is not singling out crawlers — it is refusing our test requests wholesale (a rate limit, a geo or IP rule, or a challenge page). Check what your firewall does with unfamiliar visitors before changing any bot rule.

Worth knowing Do the two tests agree?

CheckedWhether the server-robots-instructions and the server-crawl-live-test tell the same story. They often do not: robots.txt is a published promise, the live answer is what actually happens, and only the comparison shows which one to believe.

Found4 bot(s) invited by robots.txt but refused by the server — the same ones the live test above names

Worth knowingA contradiction is worth resolving whichever way you meant it. Invited-but-refused: your firewall overrules your robots.txt — align them, or accept that the file is advertising a welcome the server does not honour. Blocked-but-served: your robots.txt line is a polite request only; a bot that ignores it gets the page, so add a server rule if the block really matters.

Help Reading a point: Each point says what was checked, what was found on your site, and what to do. Faults come first; Info points are statements, not passes or fails. Read more

The G. Cloud Services group, bot by bot

✗

AliyunSecBot

robots.txt: Allowed with exceptions · your server: HTTP 403 · the server (HTTP 403)
✗

Amazonbot

robots.txt: Allowed with exceptions · your server: HTTP 403 · the server (HTTP 403)
✗

Amzn-User

robots.txt: Allowed with exceptions · your server: HTTP 403 · the server (HTTP 403)
✗

Google-CloudVertexBot

robots.txt: Allowed with exceptions · your server: HTTP 403 · the server (HTTP 403)

See the roster table

Every bot of the G. Cloud Services group, with its two verdicts
Bot	server-robots-instructions	server-crawl-live-test	What stopped it
AliyunSecBot An unusual format, but the canonical documented instance.	Allowed with exceptions by the catch-all group closed: /media-resize/, /admin/, /*/admin/ +96	HTTP 403	the server (HTTP 403)
Amazonbot Real instances wrap this inside a full desktop or mobile Chrome-style user-agent.	Allowed with exceptions by the catch-all group closed: /media-resize/, /admin/, /*/admin/ +96	HTTP 403	the server (HTTP 403)
Amzn-User	Allowed with exceptions by the catch-all group closed: /media-resize/, /admin/, /*/admin/ +96	HTTP 403	the server (HTTP 403)
Google-CloudVertexBot	Allowed with exceptions by the catch-all group closed: /media-resize/, /admin/, /*/admin/ +96	HTTP 403	the server (HTTP 403)

Help Reading the roster: One row per bot: the file's verdict, then the server's. Hover a bot's name for the user-agent we sent, and a verdict for its full detail. The last column says what actually stopped it. Read more

Other Agents

0 of 5 bots read your site.

Guide

This section answers: Which bots of this group can read your site, and what stops the others? Open guide

Words used

Gate 1
Gate 2
Captcha wall

Measured Nov. 2, 2026, 2:04 a.m.

Help Who these bots are: Archives, plagiarism checkers and the agents that fit nowhere else. Informational. The Internet Archive bots are worth a thought: they are how your old pages survive a redesign or a domain move. Read more

Your robots.txt says yes to all 5. Your server says no to 4.

5 bots knock H. Other Agents

Gate 1 Your robots.txt

5 — allowed with exceptions

All of them ride on your catch-all group (User-agent: *) — no rule in your file names one of them individually. 5 of them are allowed with exceptions — the closed paths are housekeeping only, no known content page is among them.

Gate 2 Your server, live

3 1 1

Captcha wallRefusedNot probed

4 Turned away by the server TurnitinBot HTTP 403archive.org_bot Captchaia_archiver Captchaia_archiver-web.archive.org Captcha

1 Judged on the file alone ViennaTinyBot

Their operator publishes no user-agent string, so no honest knock is possible.

Help Reading the doors: Gate 1 is your robots.txt, gate 2 your server's live answer, then the outcome each bot got. Hatched red = a captcha wall; grey = not measurable. Read more

Help Why this group is not scored: Blocking these bots is a legitimate decision about your own content: a shut door is drawn in ink, never marked wrong. Read more

What the measurement means, point by point

Worth knowing Server-robots-instructions — what robots.txt permits

CheckedThe first thing every well-behaved bot reads. A Disallow rule on the homepage stops it before it ever requests a page, so nothing further down can compensate. A group that closes some paths but not the homepage is 'allowed with exceptions' — it counts as allowed, and the exceptions are tested against your known content pages below. A bot no rule names is allowed — silence is permission — and shown as 'not indicated'.

Foundall 5 are allowed by robots.txt; 5 of them are allowed with exceptions — the closed paths are housekeeping only, no known content page is among them.

Worth knowing Server-crawl-live-test — the live HEAD request

CheckedWhat your server really answers to one HEAD request wearing that bot's name. A firewall, a CDN rule or a bot-protection product can refuse a crawler your robots.txt welcomes — and robots.txt alone can never reveal it.

Found4 forbidden: TurnitinBot, archive.org_bot, ia_archiver, ia_archiver-web.archive.org

Worth knowingRead this one before acting: an ordinary desktop browser was refused too, so your server is not singling out crawlers — it is refusing our test requests wholesale (a rate limit, a geo or IP rule, or a challenge page). Check what your firewall does with unfamiliar visitors before changing any bot rule.

Worth knowing Do the two tests agree?

CheckedWhether the server-robots-instructions and the server-crawl-live-test tell the same story. They often do not: robots.txt is a published promise, the live answer is what actually happens, and only the comparison shows which one to believe.

Found4 bot(s) invited by robots.txt but refused by the server — the same ones the live test above names

Worth knowingA contradiction is worth resolving whichever way you meant it. Invited-but-refused: your firewall overrules your robots.txt — align them, or accept that the file is advertising a welcome the server does not honour. Blocked-but-served: your robots.txt line is a polite request only; a bot that ignores it gets the page, so add a server rule if the block really matters.

Worth knowing Judged on robots.txt alone

CheckedSome operators publish a robots.txt token but never a literal user-agent string. Those bots cannot be impersonated honestly, so they are judged on your rules only — no live request was made in their name.

Found1 of 5: ViennaTinyBot

Worth knowingNothing to do. Their robots.txt verdict is still exact — it is read from your own file — only the server-side half is unknown.

Help Reading a point: Each point says what was checked, what was found on your site, and what to do. Faults come first; Info points are statements, not passes or fails. Read more

The H. Other Agents group, bot by bot

✗

TurnitinBot

robots.txt: Allowed with exceptions · your server: HTTP 403 · the server (HTTP 403)
!

ViennaTinyBot

robots.txt: Allowed with exceptions · your server: Not probed
✗

archive.org_bot

robots.txt: Allowed with exceptions · your server: Captcha · a bot wall instead of the page — redirected to a captcha page
✗

ia_archiver

robots.txt: Allowed with exceptions · your server: Captcha · a bot wall instead of the page — redirected to a captcha page
✗

ia_archiver-web.archive.org

robots.txt: Allowed with exceptions · your server: Captcha · a bot wall instead of the page — redirected to a captcha page

See the roster table

Every bot of the H. Other Agents group, with its two verdicts
Bot	server-robots-instructions	server-crawl-live-test	What stopped it
TurnitinBot A newer TurnitinBot/2.x is reported but not independently confirmed.	Allowed with exceptions by the catch-all group closed: /media-resize/, /admin/, /*/admin/ +96	HTTP 403	the server (HTTP 403)
ViennaTinyBot Only a bare lowercase identifier appears in aggregator directories.	Allowed with exceptions by the catch-all group closed: /media-resize/, /admin/, /*/admin/ +96	Not probed	—
archive.org_bot Heritrix-version-qualified variants are also documented.	Allowed with exceptions by the catch-all group closed: /media-resize/, /admin/, /*/admin/ +96	Captcha	a bot wall instead of the page — redirected to a captcha page
ia_archiver The historical Alexa string. Alexa retired in 2022; the string is kept for legacy matching.	Allowed with exceptions by the catch-all group closed: /media-resize/, /admin/, /*/admin/ +96	Captcha	a bot wall instead of the page — redirected to a captcha page
ia_archiver-web.archive.org A distinct token used by the Wayback Machine's live-web / Save Page Now fetcher.	Allowed with exceptions by the catch-all group closed: /media-resize/, /admin/, /*/admin/ +96	Captcha	a bot wall instead of the page — redirected to a captcha page

Help Reading the roster: One row per bot: the file's verdict, then the server's. Hover a bot's name for the user-agent we sent, and a verdict for its full detail. The last column says what actually stopped it. Read more

Section F · Reference

Reference

7 terms, 10 questions answered.

Guide

This section answers: What do the words mean, and what do readers ask? Open guide

The seven words this report uses

Gate 1 · server-robots-instructions: What your robots.txt says about that bot's name. Costs your server one request in total: the file is fetched once and read for all 157 bots.
Gate 2 · server-crawl-live-test: What your server answers to one live HEAD request wearing that bot's own user-agent. Headers only — no page content travels.
Allowed / Allowed with exceptions / Not allowed / Not indicated: The four robots.txt verdicts. Allowed with exceptions means the homepage is open but the governing group closes some paths — the normal shape of a well-kept file; it counts as allowed, and the closed paths are tested against your known content pages. Not allowed means the homepage itself is closed. Not indicated means no rule names the bot: by the standard, silence is permission.
Served: A normal answer to the live request — that bot gets your page.
Refused · captcha wall: An HTTP refusal (403 and its neighbours), or a success code that led to a challenge page instead of your homepage. Bots do not solve captchas, so the report counts a wall as a refusal and names it.
Rate-limited: HTTP 429: a statement about how fast our probe asked, not about who may crawl you. Excluded from every figure.
Not probed: The operator publishes a robots.txt token but no user-agent string, so no honest knock is possible. Judged on your file alone.

Frequently asked questions

What is the server-robots-instructions test?

It reads your robots.txt — the public rulebook every well-behaved bot fetches before anything else — and looks up each bot's name in it. Four answers are possible: Allowed (the governing group closes nothing), Allowed with exceptions (the homepage is open but some paths are closed — the report lists them and says whether any is a page you publish), Not allowed (the homepage itself is closed), and Not indicated (no rule names it and no catch-all exists — by the standard, silence means permission). This test costs your server one request in total: the file is fetched once and read for all 157 bots.

What is the server-crawl-live-test?

One real HTTP HEAD request to your homepage per bot, sent with that bot's own User-Agent header — the way the bot itself would knock. HEAD asks for the headers only, so no page content travels and the test costs your server almost nothing. The answer is Allowed (a normal response), Forbidden (an HTTP refusal such as 403, or a captcha wall), or Error (a timeout, a network failure or a broken 5xx answer).

What is html-meta-robots — and why is it not in this report?

html-meta-robots is the <meta name="robots" content="noindex"> tag inside a page's HTML: a third place where a site can tell bots what to do, page by page. This report never downloads your HTML (the live test is header-only by design), so it cannot read that tag — but it does read the tag's HTTP twin, the X-Robots-Tag response header, which arrives with every answer: that is the "Indexing directives in the headers" check in each group. The html-meta-robots of each promoted page is checked in the page SEO/GEO reports, which do read the HTML.

Can the two tests contradict each other?

Yes, in both directions, and the contradiction is one of the most useful findings here. Invited but refused: robots.txt allows a bot, yet the server turns its request away — a firewall or bot wall is overruling the file, possibly without anyone having decided that. Blocked but served: robots.txt forbids a bot, yet the server answers anyway — the block is a polite request that only honest bots obey. The "Do the two tests agree?" check in each group lists every case found.

Does a robots.txt block actually stop a bot?

Only if the bot chooses to obey it. robots.txt is a convention, not an enforcement: the major search engines and AI crawlers respect it scrupulously, but nothing technical prevents a scraper from ignoring the file and requesting your pages anyway. If a block really matters to you, it needs a server-side rule too — which is exactly what the server-crawl-live-test measures.

Why might Googlebot show "Forbidden" here while Google still indexes my site?

Because this test sends Googlebot's name from AkOOre's network, not from Google's. Google, Bing and Apple all recommend that sites verify crawlers by IP address or reverse DNS rather than by name — a server doing that correctly refuses anyone who merely claims to be Googlebot while serving the real one normally. That is why the report treats a refusal as a question to investigate (check Search Console or your server logs), and why the control request exists.

What does "Captcha" mean in the live-test column?

The server answered with a success code but sent the bot to a challenge page — a CAPTCHA or a JavaScript interstitial — instead of your homepage. For a crawler that is a refusal with extra steps: bots do not solve captchas, so the page behind the wall is never read. The report counts it as Forbidden and names the challenge, because a naive reading of the 200 status would have called it allowed.

What does "Not probed" mean?

Some bot operators publish a robots.txt token but never a literal user-agent string (Google-Extended and Applebot-Extended, for example, ride along inside other crawlers' requests). Those bots cannot be impersonated honestly — inventing a string would mean inventing a verdict — so they are judged on your robots.txt alone and excluded from the request percentage.

What does "Rate-limited — not measurable" mean?

Your server answered HTTP 429 ("too many requests") — a statement about how fast our probe asked, not about whether that bot is welcome. Counting it as a refusal would blame your site for our pace, so those rows are excluded from every percentage. The probe slows itself down when it sees a 429; re-running later usually measures them cleanly.

Should I block the AI bots?

Split them in two first. Citation crawlers (OAI-SearchBot, ChatGPT-User, Claude-SearchBot, PerplexityBot…) fetch pages so an assistant can quote and link you — blocking them makes you invisible in answers your customers increasingly read. Training crawlers (GPTBot, CCBot, Meta-ExternalAgent…) collect content to train models — blocking them protects your content and costs you no citations. Blocking training while allowing citation is a common, coherent middle ground; the AI Bots group shows exactly which door each bot is.

Your server is refusing requests wholesale, blocking even normal desktop browsers

Fix first — critical 4

Then improve 4

Worth knowing 18

What the measurement means, point by point

The A. Search Engines group, bot by bot

What the measurement means, point by point

The B. AI Bots group, bot by bot

What the measurement means, point by point

The C. Google Bots group, bot by bot

What the measurement means, point by point

The D. Social Bots group, bot by bot

What the measurement means, point by point

The E. SEO Tools group, bot by bot

What the measurement means, point by point

The F. Scrapers group, bot by bot

What the measurement means, point by point

The G. Cloud Services group, bot by bot

What the measurement means, point by point

The H. Other Agents group, bot by bot

The seven words this report uses

Frequently asked questions