You are exploring AkOOre's demo website — read-only, nothing you change is saved.

Who can read your site — originaluniqueart.com — generated on Nov. 16, 2026, 4:54 p.m. by AkOOre

OverviewCrawler access · originaluniqueart.com · 16 Nov 2026

Your server is refusing requests wholesale, blocking even normal desktop browsers

This report shows which bots can reach your site and which are being turned away. You have a great opportunity to open your gates by fixing a server rule that is currently blocking many visitors.

Start with section A. Work orders to see the first critical fix. Check your firewall or rate limits to stop these wholesale refusals and let the right traffic in.

Some sections are not available yet.

Introduction written 26 Oct 2026 from this report's data · the data has moved since; it is being rewritten

No
Google
Yes
Bing
No
ChatGPT
No
Perplexity
No
Claude
No
Gemini

In this report

7 sections · 157 bots · 8 groups · 39 checks
originaluniqueart.com · 16 Nov 2026

Section A · Work orders

Work orders

8 recommendations, 4 of them critical.

Fix first — critical 4

  1. Server-crawl-live-test — the live HEAD request (G. Google's own robots — this check scores 0/100)
    What to do: Read this one before acting: an ordinary desktop browser was refused too, so your server is not singling out crawlers — it is refusing our test requests wholesale (a rate limit, a geo or IP rule, or a challenge page). Check what your firewall does with unfamiliar visitors before changing any bot rule.
    What we measured on your site: 8 forbidden: Gemini-Deep-Research, Google-InspectionTool, GoogleOther, GoogleOther-Image, GoogleOther-Video, Googlebot-Image, Googlebot-Video, Storebot-Google
    Why it matters: What your server really answers to one HEAD request wearing that bot's name. A firewall, a CDN rule or a bot-protection product can refuse a crawler your robots.txt welcomes — and robots.txt alone can never reveal it.
    Where to see the detail: the G. Google's own robots section shows this check with the data it is based on.
  2. Server-crawl-live-test — the live HEAD request (F. Search engines — this check scores 15/100)
    What to do: Read this one before acting: an ordinary desktop browser was refused too, so your server is not singling out crawlers — it is refusing our test requests wholesale (a rate limit, a geo or IP rule, or a challenge page). Check what your firewall does with unfamiliar visitors before changing any bot rule.
    What we measured on your site: 17 forbidden: Amzn-SearchBot, Applebot, AspiegelBot, Baiduspider, BingPreview, Googlebot, MicrosoftPreview, MojeekBot and 9 more
    Why it matters: What your server really answers to one HEAD request wearing that bot's name. A firewall, a CDN rule or a bot-protection product can refuse a crawler your robots.txt welcomes — and robots.txt alone can never reveal it.
    Where to see the detail: the F. Search engines section shows this check with the data it is based on.
  3. Server-robots-instructions — what robots.txt permits (G. Google's own robots — this check scores 30/100)
    What to do: Your robots.txt closes the door to GoogleOther, GoogleOther-Image, GoogleOther-Video, Googlebot-Discovery, Googlebot-Image, Googlebot-News, Googlebot-Video. Open the file and look for the User-agent group named in the table below: unless the block is deliberate, delete its Disallow: / line. This is a one-line change with an outsized effect.
    What we measured on your site: 7 of 10 not allowed: GoogleOther, GoogleOther-Image, GoogleOther-Video, Googlebot-Discovery, Googlebot-Image, Googlebot-News, Googlebot-Video; 3 of them are allowed with exceptions — the closed paths…
    Why it matters: The first thing every well-behaved bot reads. A Disallow rule on the homepage stops it before it ever requests a page, so nothing further down can compensate. A group that closes some paths but not the homepage is 'allowed with exceptions' — it counts as allowed, and the exceptions are tested against your known content pages below. A bot no rule names is allowed — silence is permission — and shown as 'not indicated'.
    Where to see the detail: the G. Google's own robots section shows this check with the data it is based on.
  4. The bots that decide the outcome (G. Google's own robots — this check scores 33/100)
    What to do: Fix these before anything else in this report: each one is a whole audience that cannot see the site today. The table below names the exact rule or answer responsible for each.
    What we measured on your site: GoogleOther is stopped by its own robots.txt rule (User-agent: googleother); Googlebot-Image is stopped by its own robots.txt rule (User-agent: googlebot); your server refused every test request, a p…
    Why it matters: Not every bot in this group is worth the same. These are the ones whose verdict changes what your business gets: the indexes people actually search and the assistants they actually ask.
    Where to see the detail: the G. Google's own robots section shows this check with the data it is based on.

Worth knowing 4

  1. Do the two tests agree? (F. Search engines)
    What to do with it: A contradiction is worth resolving whichever way you meant it. Invited-but-refused: your firewall overrules your robots.txt — align them, or accept that the file is advertising a welcome the server does not honour. Blocked-but-served: your robots.txt line is a polite request only; a bot that ignores it gets the page, so add a server rule if the block really matters.
    What the data says: 16 bot(s) invited by robots.txt but refused by the server: Amzn-SearchBot, Applebot, AspiegelBot, Baiduspider, BingPreview, MicrosoftPreview, MojeekBot, PetalBot and 8 more
    Why it matters: Whether the server-robots-instructions and the server-crawl-live-test tell the same story. They often do not: robots.txt is a published promise, the live answer is what actually happens, and only the comparison shows which one to believe.
    Where to see the detail: the F. Search engines section shows this check with the data it is based on.
  2. Judged on robots.txt alone (F. Search engines)
    What to do with it: Nothing to do. Their robots.txt verdict is still exact — it is read from your own file — only the server-side half is unknown.
    What the data says: 7 of 27: Applebot-Extended, BingVideoPreview, Yahoo-Blogs, YahooSeeker, Yandex, YandexAdditionalBot, baidu
    Why it matters: Some operators publish a robots.txt token but never a literal user-agent string. Those bots cannot be impersonated honestly, so they are judged on your rules only — no live request was made in their name.
    Where to see the detail: the F. Search engines section shows this check with the data it is based on.
  3. Do the two tests agree? (G. Google's own robots)
    What to do with it: A contradiction is worth resolving whichever way you meant it. Invited-but-refused: your firewall overrules your robots.txt — align them, or accept that the file is advertising a welcome the server does not honour. Blocked-but-served: your robots.txt line is a polite request only; a bot that ignores it gets the page, so add a server rule if the block really matters.
    What the data says: 3 bot(s) invited by robots.txt but refused by the server: Gemini-Deep-Research, Google-InspectionTool, Storebot-Google
    Why it matters: Whether the server-robots-instructions and the server-crawl-live-test tell the same story. They often do not: robots.txt is a published promise, the live answer is what actually happens, and only the comparison shows which one to believe.
    Where to see the detail: the G. Google's own robots section shows this check with the data it is based on.
  4. Judged on robots.txt alone (G. Google's own robots)
    What to do with it: Nothing to do. Their robots.txt verdict is still exact — it is read from your own file — only the server-side half is unknown.
    What the data says: 2 of 10: Googlebot-Discovery, Googlebot-News
    Why it matters: Some operators publish a robots.txt token but never a literal user-agent string. Those bots cannot be impersonated honestly, so they are judged on your rules only — no live request was made in their name.
    Where to see the detail: the G. Google's own robots section shows this check with the data it is based on.

Section B · Verdict

Who can actually read your site

98 of 157 bots turned away.

157 bots tested 8 groups 126 probed live with their own user-agent last tested Nov. 16, 2026, 2:03 a.m.

Your robots.txt lets 127 of 157 in. Your server says no to 98.

157 bots knock originaluniqueart.com

Gate 1 Your robots.txt

Allowed with exceptionsNot allowed

Your file names and blocks ChatGPT Operator, ChatGPT-User, ChatGPT-User/2.0, Claude-SearchBot, Claude-User, Claude-Web, ClaudeBot, GPTBot and 22 more. 127 of them are allowed with exceptions — the closed paths are housekeeping only, no known content page is among them.

Gate 2 Your server, live

ServedCaptcha wallRefusedNot probed

28 Read your site ChatGPT-UserClaude-CodeClaude-SearchBotClaude-UserClaude-WebClaudeBotGPTBotMistralAI-User/1.0OAI-SearchBotPerplexity-User/1.0 + 18 more
98 Turned away by the server AI2Bot HTTP 403Ai2Bot-Dolma HTTP 403DeepSeekBot HTTP 403DigitalOceanGenAICrawler HTTP 403DuckAssistBot HTTP 403LINER Bot HTTP 403PanguBot HTTP 403SBIntuitionsBot HTTP 403cohere-training-data-crawler HTTP 403img2dataset HTTP 403 + 88 more
31 Judged on the file alone ChatGPT OperatorChatGPT-User/2.0Google-ExtendedGrokGrokBotMistralAI-UserMyCentralAIScraperBotPerplexity-Userquillbot.comGooglebot-Discovery + 21 more

Their operator publishes no user-agent string, so no honest knock is possible.

Control request: a plain browser

HTTP 403 — a plain browser was refused too

Sent as Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/140.0.0.0 Safari/537.36

Let them in: paste this into your robots.txt

Add these lines to the robots.txt file at the root of your website. A robot follows the group that names it and ignores the catch-all group (User-agent: *), so copy under each new group the Disallow lines of the private paths it should still skip. If your file already has a group for one of these robots, delete that group's Disallow: / line instead.

User-agent: ChatGPT-User
Allow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: Claude-User
Allow: /

User-agent: ClaudeBot
Allow: /

User-agent: GPTBot
Allow: /

User-agent: Google-Extended
Allow: /

User-agent: OAI-SearchBot
Allow: /

User-agent: Perplexity-User
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: Googlebot
Allow: /

Section C · Scores

The figures and the access standing

Critical

49/100, 6 critical checks.

49/100 Critical

05080100
Critical
6
Fix soon
3
Good
3
127/157 Allowed in robots.txt
28/126 Requests allowed
25/27 Search engines welcome
15/33 AI answer engines welcome
98 Requests forbidden

Section D · Groups

Every group, and which door it is behind

Lowest scored group: Google Bots, 21%.

Every bot group, with how many bots got each outcome
GroupScore Blocked by robots.txtServedCaptcha wallRefusedNot probed
Scored — being shut out here costs you
Search Engines 64% Fix soon 2 3 1 16 7
AI Bots 38% Critical 18 14 · 10 9
Google Bots 21% Critical 7 · 5 3 2
Social Bots 72% Fix soon 3 7 · 4 ·
Measured, never marked wrong
SEO Tools not scored · · · 20 1
Scrapers not scored · 1 1 33 11
Cloud Services not scored · · · 4 ·
Other Agents not scored · 3 · 1 1

Section E · Diagnosis

The posture we measured

108 contradictions between file and server.

Your posture

Hard wall

robots.txt welcomes the bots, but the server turns most requests away (a firewall, CDN rule or bot challenge). The published rulebook and the door policy disagree — the door wins.

What it buys you

  • Strong, enforced protection: bots that ignore robots.txt are stopped anyway.
  • Legitimate if crawler verification by IP or reverse DNS lets the real search engines through a door that name-claimers cannot use.

What it costs you

  • If the wall does NOT allowlist real crawlers by IP, search engines and AI assistants are locked out with everyone else — invisibility, not protection.
  • The welcoming robots.txt is misleading — it advertises an access the server does not grant.

Where your two rulebooks contradict each other

92 invited, then refused

robots.txt welcomes AI2Bot, Ai2Bot-Dolma, DeepSeekBot, DigitalOceanGenAICrawler, DuckAssistBot, LINER Bot and 86 more, but the server turns them away. The server wins — your file is advertising a welcome it does not honour.

16 blocked, then served

robots.txt forbids ChatGPT-User, Claude-SearchBot, Claude-User, Claude-Web, ClaudeBot, GPTBot and 10 more, yet the server serves them anyway: the block only holds against bots polite enough to honour it.

Section F · Scored group

Search engines

Fix soon

3 of 27 bots read your site.

Measured Nov. 16, 2026, 2:03 a.m.

2/3 scored points pass. Weakest point: Server-crawl-live-test — the live HEAD request (15%).

Your robots.txt lets 25 of 27 in. Your server says no to 17.

27 bots knock Search engines

Gate 1 Your robots.txt

Allowed with exceptionsNot allowed

Your file names and blocks Applebot-Extended, Googlebot. 25 of them are allowed with exceptions — the closed paths are housekeeping only, no known content page is among them.

Gate 2 Your server, live

ServedCaptcha wallRefusedNot probed

3 Read your site BingbotDuckDuckBotadidxbot
17 Turned away by the server Amzn-SearchBot HTTP 403Applebot HTTP 403AspiegelBot HTTP 403Baiduspider HTTP 403BingPreview HTTP 403MicrosoftPreview HTTP 403MojeekBot HTTP 403PetalBot HTTP 403SeznamBot HTTP 403SeznamHomepageCrawler HTTP 403 + 7 more
7 Judged on the file alone Applebot-ExtendedBingVideoPreviewYahoo-BlogsYahooSeekerYandexYandexAdditionalBotbaidu

Their operator publishes no user-agent string, so no honest knock is possible.

What the measurement means, point by point

Good Server-robots-instructions — what robots.txt permits 93%

CheckedThe first thing every well-behaved bot reads. A Disallow rule on the homepage stops it before it ever requests a page, so nothing further down can compensate. A group that closes some paths but not the homepage is 'allowed with exceptions' — it counts as allowed, and the exceptions are tested against your known content pages below. A bot no rule names is allowed — silence is permission — and shown as 'not indicated'.

Found2 of 27 not allowed: Applebot-Extended, Googlebot; 25 of them are allowed with exceptions — the closed paths are housekeeping only, no known content page is among them.

Critical Server-crawl-live-test — the live HEAD request 15%

CheckedWhat your server really answers to one HEAD request wearing that bot's name. A firewall, a CDN rule or a bot-protection product can refuse a crawler your robots.txt welcomes — and robots.txt alone can never reveal it.

Found17 forbidden: Amzn-SearchBot, Applebot, AspiegelBot, Baiduspider, BingPreview, Googlebot, MicrosoftPreview, MojeekBot and 9 more

Do thisRead this one before acting: an ordinary desktop browser was refused too, so your server is not singling out crawlers — it is refusing our test requests wholesale (a rate limit, a geo or IP rule, or a challenge page). Check what your firewall does with unfamiliar visitors before changing any bot rule.

Worth knowing Indexing directives in the headers

CheckedThe X-Robots-Tag response header of the answers your server actually served. A noindex here lets the bot in and then tells it to throw the page away — the most expensive way to be invisible, because it looks fine in every other check. Only served answers are read: a wall's or an error page's header is not yours.

Foundno X-Robots-Tag header in any served answer

Worth knowing Do the two tests agree?

CheckedWhether the server-robots-instructions and the server-crawl-live-test tell the same story. They often do not: robots.txt is a published promise, the live answer is what actually happens, and only the comparison shows which one to believe.

Found16 bot(s) invited by robots.txt but refused by the server: Amzn-SearchBot, Applebot, AspiegelBot, Baiduspider, BingPreview, MicrosoftPreview, MojeekBot, PetalBot and 8 more

Worth knowingA contradiction is worth resolving whichever way you meant it. Invited-but-refused: your firewall overrules your robots.txt — align them, or accept that the file is advertising a welcome the server does not honour. Blocked-but-served: your robots.txt line is a polite request only; a bot that ignores it gets the page, so add a server rule if the block really matters.

Good The bots that decide the outcome 83%

CheckedNot every bot in this group is worth the same. These are the ones whose verdict changes what your business gets: the indexes people actually search and the assistants they actually ask.

FoundGooglebot is stopped by its own robots.txt rule (User-agent: googlebot); your server refused every test request, a plain browser's included, so its answer to Applebot, Baiduspider, YandexBot is not counted here

Worth knowing Judged on robots.txt alone

CheckedSome operators publish a robots.txt token but never a literal user-agent string. Those bots cannot be impersonated honestly, so they are judged on your rules only — no live request was made in their name.

Found7 of 27: Applebot-Extended, BingVideoPreview, Yahoo-Blogs, YahooSeeker, Yandex, YandexAdditionalBot, baidu

Worth knowingNothing to do. Their robots.txt verdict is still exact — it is read from your own file — only the server-side half is unknown.

The Search engines group, bot by bot

  1. ✓

    adidxbot

    robots.txt: Allowed with exceptions · your server: Redirected

  2. ✗

    Amzn-SearchBot

    robots.txt: Allowed with exceptions · your server: HTTP 403

  3. ✗

    Applebot

    robots.txt: Allowed with exceptions · your server: HTTP 403

  4. !

    Applebot-Extended

    robots.txt: Not allowed · your server: Not probed · its own robots.txt rule (User-agent: applebot-extended)

  5. ✗

    AspiegelBot

    robots.txt: Allowed with exceptions · your server: HTTP 403

  6. !

    baidu

    robots.txt: Allowed with exceptions · your server: Not probed

  7. ✗

    Baiduspider

    robots.txt: Allowed with exceptions · your server: HTTP 403

  8. ✓

    Bingbot

    robots.txt: Allowed with exceptions · your server: Redirected

  9. ✗

    BingPreview

    robots.txt: Allowed with exceptions · your server: HTTP 403

  10. !

    BingVideoPreview

    robots.txt: Allowed with exceptions · your server: Not probed

  11. ✓

    DuckDuckBot

    robots.txt: Allowed with exceptions · your server: Redirected

  12. ✗

    Googlebot

    robots.txt: Not allowed · your server: Captcha · its own robots.txt rule (User-agent: googlebot)

  13. ✗

    MicrosoftPreview

    robots.txt: Allowed with exceptions · your server: HTTP 403

  14. ✗

    MojeekBot

    robots.txt: Allowed with exceptions · your server: HTTP 403

  15. ✗

    PetalBot

    robots.txt: Allowed with exceptions · your server: HTTP 403

  16. ✗

    SeznamBot

    robots.txt: Allowed with exceptions · your server: HTTP 403

  17. ✗

    SeznamHomepageCrawler

    robots.txt: Allowed with exceptions · your server: HTTP 403

  18. ✗

    Slurp

    robots.txt: Allowed with exceptions · your server: HTTP 403

  19. ✗

    Teoma

    robots.txt: Allowed with exceptions · your server: HTTP 403

  20. !

    Yahoo-Blogs

    robots.txt: Allowed with exceptions · your server: Not probed

  21. ✗

    Yahoo-FeedSeeker

    robots.txt: Allowed with exceptions · your server: HTTP 403

  22. ✗

    Yahoo-MMCrawler

    robots.txt: Allowed with exceptions · your server: HTTP 403

  23. !

    YahooSeeker

    robots.txt: Allowed with exceptions · your server: Not probed

  24. !

    Yandex

    robots.txt: Allowed with exceptions · your server: Not probed

  25. ✗

    YandexAdditional

    robots.txt: Allowed with exceptions · your server: HTTP 403

  26. !

    YandexAdditionalBot

    robots.txt: Allowed with exceptions · your server: Not probed

  27. ✗

    YandexBot

    robots.txt: Allowed with exceptions · your server: HTTP 403

See the roster table
Every bot of the Search engines group, with its two verdicts
Bot Gate 1 · your robots.txt Gate 2 · your server, live What stopped it
adidxbot
Bing's ad-landing-page quality crawler.
Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +99
Redirected —
Amzn-SearchBot
Feeds Amazon, Alexa and Rufus search.
Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +99
HTTP 403 the server (HTTP 403)
Applebot
The device, OS and WebKit fields vary between Apple's crawlers.
Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +99
HTTP 403 the server (HTTP 403)
Applebot-Extended
A robots.txt token with no distinct user-agent: it governs whether Apple Intelligence may train on content Applebot already fetched.
Not allowed
by User-agent: applebot-extended
Not probed its own robots.txt rule (User-agent: applebot-extended)
AspiegelBot
alias — same bot as another row
The legacy name, superseded by PetalBot.
Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +99
HTTP 403 the server (HTTP 403)
baidu
alias — same bot as another row
The same bot as Baiduspider, not a separate name or string.
Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +99
Not probed —
Baiduspider Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +99
HTTP 403 the server (HTTP 403)
Bingbot Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +99
Redirected —
BingPreview Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +99
HTTP 403 the server (HTTP 403)
BingVideoPreview
Bing confirms this bot exists but publishes no literal user-agent string, so only its robots.txt token is checked.
Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +99
Not probed —
DuckDuckBot Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +99
Redirected —
Googlebot
The mobile variant wraps a Chrome/Android string around the same (compatible; Googlebot/2.1; ...) suffix.
Not allowed
by User-agent: googlebot
Captcha its own robots.txt rule (User-agent: googlebot)
MicrosoftPreview Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +99
HTTP 403 the server (HTTP 403)
MojeekBot
The version number increments over time (0.2 to 0.11+ seen).
Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +99
HTTP 403 the server (HTTP 403)
PetalBot
Huawei Petal Search. The mobile variant prefixes Android and AppleWebKit fields.
Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +99
HTTP 403 the server (HTTP 403)
SeznamBot Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +99
HTTP 403 the server (HTTP 403)
SeznamHomepageCrawler Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +99
HTTP 403 the server (HTTP 403)
Slurp Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +99
HTTP 403 the server (HTTP 403)
Teoma Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +99
HTTP 403 the server (HTTP 403)
Yahoo-Blogs
Legacy, listed only by unofficial aggregators.
Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +99
Not probed —
Yahoo-FeedSeeker
Historical form, not verified as current.
Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +99
HTTP 403 the server (HTTP 403)
Yahoo-MMCrawler
Historical form, not verified as current.
Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +99
HTTP 403 the server (HTTP 403)
YahooSeeker
Legacy: Yahoo Search is Bing-powered now and no current documentation publishes a string.
Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +99
Not probed —
Yandex
alias — same bot as another row
A robots.txt control token matching any Yandex* bot, not a distinct request user-agent.
Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +99
Not probed —
YandexAdditional
Mostly a robots.txt control token; this string comes from secondary sources, not from Yandex's own documentation.
Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +99
HTTP 403 the server (HTTP 403)
YandexAdditionalBot
alias — same bot as another row
Same bot and same token as YandexAdditional.
Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +99
Not probed —
YandexBot Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +99
HTTP 403 the server (HTTP 403)

Section G · Scored group

Google's own robots

Critical

0 of 10 bots read your site.

Measured Nov. 16, 2026, 2:03 a.m.

0/3 scored points pass. Weakest point: Server-crawl-live-test — the live HEAD request (0%).

Your robots.txt lets 3 of 10 in. Your server says no to 8.

10 bots knock Google's own robots

Gate 1 Your robots.txt

Allowed with exceptionsNot allowed

Your file names and blocks GoogleOther, GoogleOther-Image, GoogleOther-Video, Googlebot-Discovery, Googlebot-Image, Googlebot-News, Googlebot-Video. 3 of them are allowed with exceptions — the closed paths are housekeeping only, no known content page is among them.

Gate 2 Your server, live

Captcha wallRefusedNot probed

8 Turned away by the server Gemini-Deep-Research HTTP 403Google-InspectionTool HTTP 403Storebot-Google HTTP 403GoogleOther CaptchaGoogleOther-Image CaptchaGoogleOther-Video CaptchaGooglebot-Image CaptchaGooglebot-Video Captcha
2 Judged on the file alone Googlebot-DiscoveryGooglebot-News

Their operator publishes no user-agent string, so no honest knock is possible.

What the measurement means, point by point

Critical Server-robots-instructions — what robots.txt permits 30%

CheckedThe first thing every well-behaved bot reads. A Disallow rule on the homepage stops it before it ever requests a page, so nothing further down can compensate. A group that closes some paths but not the homepage is 'allowed with exceptions' — it counts as allowed, and the exceptions are tested against your known content pages below. A bot no rule names is allowed — silence is permission — and shown as 'not indicated'.

Found7 of 10 not allowed: GoogleOther, GoogleOther-Image, GoogleOther-Video, Googlebot-Discovery, Googlebot-Image, Googlebot-News, Googlebot-Video; 3 of them are allowed with exceptions — the closed paths are housekeeping only, no known content page is among them.

Do thisYour robots.txt closes the door to GoogleOther, GoogleOther-Image, GoogleOther-Video, Googlebot-Discovery, Googlebot-Image, Googlebot-News, Googlebot-Video. Open the file and look for the User-agent group named in the table below: unless the block is deliberate, delete its Disallow: / line. This is a one-line change with an outsized effect.

Critical Server-crawl-live-test — the live HEAD request 0%

CheckedWhat your server really answers to one HEAD request wearing that bot's name. A firewall, a CDN rule or a bot-protection product can refuse a crawler your robots.txt welcomes — and robots.txt alone can never reveal it.

Found8 forbidden: Gemini-Deep-Research, Google-InspectionTool, GoogleOther, GoogleOther-Image, GoogleOther-Video, Googlebot-Image, Googlebot-Video, Storebot-Google

Do thisRead this one before acting: an ordinary desktop browser was refused too, so your server is not singling out crawlers — it is refusing our test requests wholesale (a rate limit, a geo or IP rule, or a challenge page). Check what your firewall does with unfamiliar visitors before changing any bot rule.

Worth knowing Do the two tests agree?

CheckedWhether the server-robots-instructions and the server-crawl-live-test tell the same story. They often do not: robots.txt is a published promise, the live answer is what actually happens, and only the comparison shows which one to believe.

Found3 bot(s) invited by robots.txt but refused by the server: Gemini-Deep-Research, Google-InspectionTool, Storebot-Google

Worth knowingA contradiction is worth resolving whichever way you meant it. Invited-but-refused: your firewall overrules your robots.txt — align them, or accept that the file is advertising a welcome the server does not honour. Blocked-but-served: your robots.txt line is a polite request only; a bot that ignores it gets the page, so add a server rule if the block really matters.

Critical The bots that decide the outcome 33%

CheckedNot every bot in this group is worth the same. These are the ones whose verdict changes what your business gets: the indexes people actually search and the assistants they actually ask.

FoundGoogleOther is stopped by its own robots.txt rule (User-agent: googleother); Googlebot-Image is stopped by its own robots.txt rule (User-agent: googlebot); your server refused every test request, a plain browser's included, so its answer to Google-InspectionTool is not counted here

Do thisFix these before anything else in this report: each one is a whole audience that cannot see the site today. The table below names the exact rule or answer responsible for each.

Worth knowing Judged on robots.txt alone

CheckedSome operators publish a robots.txt token but never a literal user-agent string. Those bots cannot be impersonated honestly, so they are judged on your rules only — no live request was made in their name.

Found2 of 10: Googlebot-Discovery, Googlebot-News

Worth knowingNothing to do. Their robots.txt verdict is still exact — it is read from your own file — only the server-side half is unknown.

The Google's own robots group, bot by bot

  1. ✗

    Gemini-Deep-Research

    robots.txt: Allowed with exceptions · your server: HTTP 403

  2. ✗

    Google-InspectionTool

    robots.txt: Allowed with exceptions · your server: HTTP 403

  3. !

    Googlebot-Discovery

    robots.txt: Not allowed · your server: Not probed · its own robots.txt rule (User-agent: googlebot)

  4. ✗

    Googlebot-Image

    robots.txt: Not allowed · your server: Captcha · its own robots.txt rule (User-agent: googlebot)

  5. !

    Googlebot-News

    robots.txt: Not allowed · your server: Not probed · its own robots.txt rule (User-agent: googlebot)

  6. ✗

    Googlebot-Video

    robots.txt: Not allowed · your server: Captcha · its own robots.txt rule (User-agent: googlebot)

  7. ✗

    GoogleOther

    robots.txt: Not allowed · your server: Captcha · its own robots.txt rule (User-agent: googleother)

  8. ✗

    GoogleOther-Image

    robots.txt: Not allowed · your server: Captcha · its own robots.txt rule (User-agent: googleother)

  9. ✗

    GoogleOther-Video

    robots.txt: Not allowed · your server: Captcha · its own robots.txt rule (User-agent: googleother)

  10. ✗

    Storebot-Google

    robots.txt: Allowed with exceptions · your server: HTTP 403

See the roster table
Every bot of the Google's own robots group, with its two verdicts
Bot Gate 1 · your robots.txt Gate 2 · your server, live What stopped it
Gemini-Deep-Research Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +99
HTTP 403 the server (HTTP 403)
Google-InspectionTool
What Search Console's URL Inspection uses. A mobile variant wraps it in a Nexus 5X Chrome string.
Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +99
HTTP 403 the server (HTTP 403)
Googlebot-Discovery
alias — same bot as another row
No documentation found for this exact name. Discover surfacing uses the standard Googlebot / Googlebot-Image, so this is likely a naming confusion.
Not allowed
by User-agent: googlebot
Not probed its own robots.txt rule (User-agent: googlebot)
Googlebot-Image Not allowed
by User-agent: googlebot
Captcha its own robots.txt rule (User-agent: googlebot)
Googlebot-News
alias — same bot as another row
A distinct robots.txt token with no separate string: the requests use the standard Googlebot user-agent.
Not allowed
by User-agent: googlebot
Not probed its own robots.txt rule (User-agent: googlebot)
Googlebot-Video Not allowed
by User-agent: googlebot
Captcha its own robots.txt rule (User-agent: googlebot)
GoogleOther
A bare token; may also appear via standard Googlebot-style strings.
Not allowed
by User-agent: googleother
Captcha its own robots.txt rule (User-agent: googleother)
GoogleOther-Image Not allowed
by User-agent: googleother
Captcha its own robots.txt rule (User-agent: googleother)
GoogleOther-Video Not allowed
by User-agent: googleother
Captcha its own robots.txt rule (User-agent: googleother)
Storebot-Google
An Android/mobile variant also exists.
Allowed with exceptions
by the catch-all group
closed: /media-resize/, /admin/, /*/admin/ +99
HTTP 403 the server (HTTP 403)

Add to your ToDo list

Send feedback