Do these 3 things before closing this tab:
1Fix the driver behind crashes, sound loss and screen glitches2Clear out junk files and repair common Windows errors3Scan for outdated or missing drivers - takes under a minuteiTechGuides is reader-supported. When you buy through links on our site, we may earn an affiliate commission. As an Amazon Associate I earn from qualifying purchases. Learn more
Robots.txt is still a crawler-access protocol, but some providers now expose separate controls for separate uses of website content. With OpenAI, for example, a site can allow its search crawler while disallowing its training crawler. Google’s Google-Extended token is a different kind of control: it addresses specified Gemini uses and does not change Google Search inclusion or ranking.
What robots.txt does—and what it cannot do
A robots.txt file communicates which parts of a site a crawler is requested to access. It is not an access-control system: the IETF’s RFC 9309 states, “These rules are not a form of access authorization.” The file is publicly discoverable, and a crawler may ignore rules.
| # | Preview | Product | Price | |
|---|---|---|---|---|
| 1 |
|
Advanced Robots.txt Generator Manual | $32.46 | Buy on Amazon |
A Disallow rule can keep a compliant crawler from fetching a matching URL, but blocking a fetch is not the same as removing a page from search results. Google says a blocked URL may still appear in Search if other pages link to it. To keep a page out of Google Search, Google recommends a supported noindex directive while allowing Googlebot to crawl the page, password protection, or removing the page. If robots.txt blocks the page, Googlebot cannot fetch it to see its noindex instruction. See Google’s robots.txt introduction.
Quick wins for a faster PC:
Scan for outdated or missing drivers - takes under a minuteDriver Scan →Repair Windows errors before they cause bigger problemsFix Now →For confidential material, use authentication or another real security control. Robots.txt is a crawler preference, not a barrier to access.
#1 Best Overall
OpenAI: choose separately for search and possible training
OpenAI documents two distinct crawler controls. OAI-SearchBot is used to surface websites in ChatGPT search features; GPTBot crawls content that may be used to train OpenAI foundation models. OpenAI says, “Each setting is independent of the others.” A site can therefore allow OAI-SearchBot and disallow GPTBot, or make the opposite choice, according to its policy.
OpenAI says a robots.txt update can take about 24 hours to adjust its search systems. That is OpenAI’s stated timing for its systems, not a general guarantee that every crawler will apply a change on the same schedule.
Google: Google-Extended is a use control, not a second crawler
Google-Extended is a robots.txt control token for whether content Google crawls may be used for specified Gemini training and grounding purposes. It is not a separate HTTP user-agent string; Google says its crawlers continue to use existing Google user-agent strings. Google also states that Google-Extended “does not impact a site’s inclusion in Google Search nor is it used as a ranking signal in Google Search.” See Google’s crawler documentation.
This differs from OpenAI’s independently configurable search and training crawlers. Do not assume another provider offers the same controls, or that a similarly named token has the same effect. Check each provider’s documentation for the specific crawler, use, and stated behavior.
Choose the control that matches your goal
| Goal | Relevant control | What it does—and does not do |
|---|---|---|
| Allow or limit ChatGPT search discovery | OpenAI’s OAI-SearchBot rule | Controls that crawler’s requested access for ChatGPT search features; it is separate from GPTBot’s setting. |
| Indicate that content should not be used for OpenAI foundation-model training | OpenAI’s GPTBot rule | Controls GPTBot access associated with content that may be used in training; it does not, by itself, set the OAI-SearchBot rule. |
| Control specified Gemini training and grounding uses | Google-Extended | Controls those specified uses; it does not affect Google Search inclusion or ranking. |
| Remove a page from Google Search | Noindex, password protection, or page removal | A robots.txt block alone does not ensure search-result removal. Googlebot must be able to fetch a page to read its noindex directive. |
| Keep content private | Authentication or another access-control mechanism | Robots.txt is not authorization and cannot make a publicly accessible page private. |
Publish rules at the right scope
RFC 9309 specifies that the file belongs at the site’s top-level /robots.txt path, uses UTF-8, and is served as text/plain. Google’s rules apply only to the host, protocol, and port serving that file. A policy at https://example.com/robots.txt does not automatically govern a subdomain or the HTTP version of the site. See Google’s robots.txt specification.
Rules are grouped by crawler product token. Under RFC 9309, matching is case-insensitive, matching groups are combined, and a wildcard group applies when no specific group matches. Google additionally documents its own path-matching behavior, including support for * and $; do not assume that every crawler implements all of Google’s matching details.
Verify changes and account for crawler differences
- Check the exact file served at the relevant host, protocol, and port—not just a local copy or the site’s main domain.
- Confirm that each intended provider token appears in the applicable group and that the rule expresses the intended access decision.
- RFC 9309 says crawlers should not use a cached robots.txt for more than 24 hours unless it is unreachable. Google says it generally caches the file for up to 24 hours and may cache longer if it cannot refresh it.
- Do not assume identical error handling across crawlers. RFC 9309 permits access when robots.txt is unavailable with a 4xx status, while an unreachable file caused by server or network errors calls for crawlers to assume complete disallow. Google documents its own handling, including 5xx and caching behavior.
- For OpenAI search systems, allow about 24 hours after a robots.txt change according to OpenAI’s stated timing, then verify against the live file and the provider’s current documentation.
Maintain a policy record for each provider that names the intended use, the documented token or directive, the file’s scope, and how the rule will be checked. Neither a Disallow rule nor a later policy change should be treated as proof that content already crawled has been removed or that behavior beyond the provider’s documented statement is guaranteed.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
