The Tool Desk
Outbyte Driver Updater FREEScan for outdated or missing drivers - takes under a minuteDriver Scan →Outbyte PC Repair FREERepair Windows errors before they cause bigger problemsFix Now →iTechGuides is reader-supported. When you buy through links on our site, we may earn an affiliate commission. As an Amazon Associate I earn from qualifying purchases. Learn more
To ask a compliant AI crawler not to access your whole site, publish a robots.txt file at the root of the relevant host with that crawler’s documented user-agent token followed by Disallow: /. This is a crawl request, not a security barrier: it cannot make public pages private or guarantee that a crawler will obey.
How to block a specific AI crawler
Use the exact user-agent token documented by the crawler’s operator. For example, this rule asks GPTBot not to crawl any path on the applicable host:
User-agent: GPTBot
Disallow: /Anthropic gives the same pattern for ClaudeBot:
User-agent: ClaudeBot
Disallow: /Each crawler needs a matching rule group if you want to ask multiple crawlers to stay away. To restrict a crawler only from selected paths, replace / with the path you want to disallow; consult the operator’s guidance for how that crawler interprets rules. The Robots Exclusion Protocol defines user-agent product tokens and Allow/Disallow rules. Google’s parser selects the most specific matching user-agent group for Google crawlers, so do not assume every crawler handles rule groups identically. IETF RFC 9309 · Google’s robots.txt specification
Put the file in the right place and verify it
Google’s requirements are useful practical guidance: robots.txt must be a UTF-8 text file at the root of the applicable site host. Its scope is limited to the protocol, host, and port where it is published. A file for https://example.com does not automatically cover https://www.example.com, another subdomain, a different port, or the HTTP version of the host. Publish a file for every host and protocol where you want the rule to apply. Google Search Central: Robots.txt introduction and guide · Google: Create and submit a robots.txt file
- Create a plain UTF-8 text file named
robots.txtwith the crawler’s user-agent token and the applicableDisallowpath. - Place it at the root of each applicable host, such as
https://example.com/robots.txtand, if needed,https://www.example.com/robots.txt. - Visit each published robots.txt URL directly. Confirm the file loads and its contents match the intended host and rules.
- Test the rules with the relevant crawler-specific tools or parser guidance where available, and check whether a CDN, firewall, authentication layer, or server configuration separately changes access. Google notes that access to the site root may require help from the hosting provider. Google’s file creation and testing guidance
Anthropic specifically advises placing the file in the top-level directory and repeating the opt-out on every subdomain where it is wanted. It also documents Crawl-delay as a non-standard extension; do not rely on that directive as a universal control. Anthropic Help Center
Choose crawler rules by what you want to allow
AI companies may operate separate crawlers for model training, search, and user-requested fetching. Blocking one token does not necessarily block the others, and can affect how the service discovers or retrieves your pages.
| Operator and token | Documented role | Effect to consider |
|---|---|---|
OpenAI — GPTBot |
Content that may be used to train OpenAI’s generative AI foundation models. | Disallowing it addresses this crawler; it does not by itself block OpenAI’s other documented crawlers. |
OpenAI — OAI-SearchBot |
Surfaces websites in ChatGPT search features. | Its setting is independent of GPTBot’s, so consider separately whether you want this search access. |
OpenAI — ChatGPT-User |
User-triggered fetches. | OpenAI says robots.txt rules may not apply because visits are initiated by user actions. |
Anthropic — ClaudeBot |
Content that could contribute to model training. | Anthropic documents it separately from its user-directed and search crawlers. |
Anthropic — Claude-User |
User-directed retrieval. | Disabling it may affect user-directed retrieval. |
Anthropic — Claude-SearchBot |
Search result quality. | Disabling it may affect search visibility. |
These roles and consequences are described in the operators’ documentation; their behavior and policies can change. Review the current pages before choosing rules: OpenAI crawler overview · Anthropic crawler guidance. Do not treat a provider’s crawler names as interchangeable.
What robots.txt cannot prevent
It cannot enforce access restrictions
The IETF standard states: “These rules are not a form of access authorization.” Google likewise explains that crawlers are expected to obey robots.txt but the file cannot enforce their behavior; some crawlers may not support or follow it. A disallow rule therefore reduces requests only from crawlers that honor the instruction. RFC 9309 · Google’s robots.txt guide
It cannot keep public pages secret or guarantee removal from search
A URL disallowed from crawling can still be discovered through links and may appear in Google Search results. The result can expose the URL and other publicly available information, such as anchor text, even when Google could not crawl the page body. Blocking crawling is not a reliable way to remove a URL from search or conceal its contents.
- For private material: require authentication, use password protection, or apply server-side access controls.
- For search visibility: use an indexing control or removal process suited to the goal. An on-page
noindexdirective only helps a crawler that can access the page and read it.
Google explains the distinction between crawl blocking, indexing, and security.
Quick Recap
Match the control to the goal
| Your goal | Approach | Important limit |
|---|---|---|
| Reduce requests from compliant AI crawlers | Publish crawler-specific robots.txt rules for the paths you want disallowed. |
The crawler must honor the request. |
| Prevent access to private content | Enforce authentication, password protection, or server-side access controls. | robots.txt does not authorize or block users. |
| Control whether a page appears in search | Use indexing controls or the relevant search-engine removal process. | A crawl-disallowed URL may remain indexed; a crawler needs access to read an on-page noindex. |
| Keep some AI access while limiting other uses | Allow or disallow each provider’s documented crawler tokens according to their separate purposes. | Rules and consequences vary by provider and crawler. |
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
Do these 3 things before closing this tab:
1Fix the driver behind crashes, sound loss and screen glitches2Clear out junk files and repair common Windows errors3Scan for outdated or missing drivers - takes under a minute
