AI crawlers can read a public page only if they can reach it and your site’s controls let the request through. A permissive robots.txt file does not prove that a crawler can fetch a page: a CDN, firewall, login requirement, or server error can still block it. To check access, identify the crawler and its purpose, inspect the robots.txt served for the exact host, request a representative page, and verify the result in your logs or edge-security events.
This is a practical check rather than a report of a specific site test: no site address or request results are available here, so there is no basis to claim that a particular site is readable or blocked.
“AI crawler” can mean several different bots
There is no single AI crawler, and the bot’s purpose affects which control matters. Search retrieval, model training, and a page fetch requested by a person are distinct activities. Check the operator’s documentation for the current behavior of each bot before writing a rule.
| Operator and token | Documented purpose | What to know when checking access |
|---|---|---|
| Googlebot | Crawls pages for Google Search and related Search features. | Google’s guidance for AI features in Search points to Googlebot directives and preview controls. Google’s AI features documentation describes controls including nosnippet, data-nosnippet, max-snippet, and noindex. |
| Google-Extended | A robots.txt token controlling specified use of content Google crawls for future Gemini model training and grounding in Gemini Apps and Vertex AI. | Google says Google-Extended is not a separate HTTP request user agent. Its use does not affect inclusion in Google Search or act as a Search ranking signal; it is not the control for Google Search AI features. See Google’s common crawlers reference and Google’s crawling guidance. |
| GPTBot | OpenAI crawling for content that may be used to train foundation models. | Use this token when considering the documented training-related crawl purpose, not as a generic label for every OpenAI request. |
| OAI-SearchBot | OpenAI crawling associated with ChatGPT search. | Its purpose differs from GPTBot; decide separately whether to allow it. |
| ChatGPT-User | Fetches pages in response to user actions. | It is not used for automatic web crawling, and robots.txt rules may not apply to user-initiated visits. OpenAI documents these distinctions in its bot reference. |
| Other providers | Examples include ClaudeBot, Claude-SearchBot, Claude-User, and PerplexityBot. | Cloudflare’s verified-bots reference is a useful inventory, but consult each operator’s documentation for its own policies and verification methods. |
These names describe declared roles, not proof that a given request came from that operator. User-agent strings can be spoofed, so use an operator’s published verification guidance or IP information when identity matters.
Outdated Drivers Are Slowing You Down
One free scan finds every outdated or missing driver and matches the right update for your exact hardware.Free scan · exact hardware matchWindows Errors? Fix Them Before They Spread
Repair common Windows errors and clear accumulated junk for a smoother, more stable PC - no reinstall needed.Free scan · no reinstall#1 Best Overall
How to check whether a crawler can reach your site
- Decide what you mean by “read.” Is the question about ordinary search visibility, AI search results, content use for model training, or a page fetch triggered by a user? Select the relevant operator and token; Google and OpenAI document different controls for different purposes.
- Inspect robots.txt on the exact site address. Check the protocol and hostname people use, such as
https://example.com/robots.txtandhttps://www.example.com/robots.txt. Google says a robots.txt file applies only to the host, protocol, and port where it is served; a rule on one host does not automatically apply to another. Read the applicable crawler-specific group as well as the wildcard group. See Google’s robots.txt documentation. - Request a representative public page. Record what happens: a successful response, redirect, denial, challenge, or unavailable page. A robots.txt
Allowrule does not override a 403 response or other block from a web application firewall (WAF), CDN, authentication layer, or origin server. - Check server and edge logs. Look for requests to the page, timestamps, status codes, and the layer that handled the request. Confirm the crawler’s identity using its operator’s published verification method where available; a matching user-agent alone is not reliable evidence.
- Use a control that matches your goal. Use robots.txt to express crawl preferences to compliant bots, authentication to restrict private material, and Google’s supported noindex controls when the goal is to prevent Google Search indexing. For Googlebot to see a page-level noindex directive, the page must remain crawlable.
- Recheck after a change. A rule change does not guarantee an immediate effect. Google says some changes to Search preview controls can take days to months to be recrawled and processed. Timing depends on the crawler and its revisit schedule.
What robots.txt can—and cannot—tell you
It expresses a preference, not a security boundary
Robots.txt is publicly accessible and is a request to crawlers that follow its rules, not technical access control. Google warns that a disallowed URL may still be discovered and indexed without its contents being crawled. Keep confidential pages behind authentication rather than relying on a disallow rule. See Google’s explanation of robots.txt and indexing.
Disallow is not the same as deindex
A robots.txt block can prevent a compliant crawler from reading page content, but it does not guarantee that the URL will disappear from search results. Conversely, a noindex directive cannot be read if robots.txt prevents Googlebot from fetching the page. If preventing Google Search indexing is the goal, allow Googlebot to crawl the page so it can see the supported noindex directive.
Google-Extended does not control all Google AI features
Google documents Google-Extended as a control over specified uses of crawled content for Gemini, not as a way to remove pages from Search or control AI Overviews and AI Mode. For Google’s AI features in Search, use the applicable Googlebot directives and preview controls described in its AI features guidance.
Why an allowed page may still be blocked
Robots.txt and actual reachability are separate checks. A crawler may be permitted by the published rules but receive a challenge or denial from a security product, fail authentication, or encounter a server problem. OpenAI advises site owners to check web-protection systems for false-positive 403 blocks. Its publishers and developers FAQ describes that diagnostic path. Cloudflare also documents separate bot controls in its bot reference and managed robots.txt documentation.
The Tool Desk
Outbyte Driver Updater FREEScan for outdated or missing drivers - takes under a minuteDriver Scan →Outbyte PC Repair FREERepair Windows errors before they cause bigger problemsFix Now →When the result is a 403, challenge page, or timeout, use the logs to find which layer issued it, then review that layer’s rules. Changing robots.txt alone will not fix a firewall or origin-server denial.
Independent reader supportYour contribution helps us test, update, and keep practical guides available for everyone.What a reliable check can establish
- Robots.txt check: what instructions a compliant crawler can see at a particular host, protocol, and port.
- Page-request check: whether a request to a particular URL received a successful response, redirect, denial, challenge, or failure at the time tested.
- Log and identity check: whether requests attributed to a crawler reached your infrastructure and what status they received, subject to reliable identity verification.
No single result proves that every AI service can or cannot read every page. A useful conclusion names the crawler, host, URL, time, observed response, and relevant security layer rather than making a blanket claim about “AI.”
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
