iTechGuides is reader-supported. When you buy through links on our site, we may earn an affiliate commission. As an Amazon Associate I earn from qualifying purchases. Learn more
You can block AI training crawlers without blocking search engines by setting separate rules for each crawler in your site’s robots.txt file. Keep Googlebot and any AI search crawler you want to allow, disallow training crawlers you want to exclude, and decide separately whether to allow user-initiated fetches. Then check for additional CDN or web application firewall (WAF) rules. Robots.txt is a crawler preference, not a security barrier: protect private pages with authentication, and use noindex when your goal is to keep a page out of Google results.
Start by choosing the outcome you want
“Block AI agents” can mean several different things. Before changing crawler rules, decide whether you want to reduce automated requests, prevent a particular crawler from collecting material for training, keep pages out of search results, or make content inaccessible to anyone without permission. Those goals need different controls.
| Goal | Control to consider | What it does—and does not do |
|---|---|---|
| Ask a compliant crawler not to fetch some or all pages | robots.txt |
Publishes crawl instructions for crawlers that honor them. It does not enforce access restrictions. |
| Keep a page out of Google results | noindex |
Signals that Google should not index the page. Google must be able to crawl it to see the directive. |
| Keep content private | Authentication or another server-side access control | Restricts access rather than relying on a crawler to follow a preference. |
| Apply policy at a CDN or WAF | Provider-specific bot controls and WAF rules | Adds an enforcement or detection layer separate from the origin’s robots.txt. Check the effective rules for your site. |
Google says robots.txt can manage crawl traffic but should not be used to hide pages from search results. A blocked URL can still appear in results, for example if Google learns about it from other pages. If a page must not be publicly accessible, require authentication instead of relying on whether a crawler follows a directive.
Free tools Windows power users keep installed
One-click scans. No signup required.
These controls address different points in the process. Robots.txt communicates which URLs a compliant crawler is asked to fetch. A noindex directive concerns whether Google includes a crawlable page in results. Authentication or another access control determines whether a request can retrieve protected content at all. A CDN or WAF can add its own bot policy before a request reaches the origin. Choose by desired outcome, not by the label “AI bot blocker.”
#1 Best Overall
- 【◆Powerful Celeron N2840 Processor: N2840 Processor, 2 Cores 2 Threads, 1M Cache, Max Turbo Frequency 2.58 GHz, TDP 7.5 W. Compatible with OPNsense, Linux, Windows,ESXI, OpenWrt and other systems. Press "Delete" key to enter BIOS setup, supports Auto Power On, Wake On Lake, GPIO, PXE
- 【◆1GbE LAN: Mini Router PC with 2*Realtek RTL8111H network card chip full UDE 1000M with filter connector.Soft Router can monitor network data, improve network security, powerful and widely used.
- ◆DDR3L Memory & Large Storage Capacity: Firewall box computer with 1 x DDR3L SO-DIMM memory 1333/1600MHz, 1xMSATA3.0 SSD+1x2.5''SATA3.0 SSD/HDD.
- ◆UHD Graphics & Dual Display: N2840 processor integrated UHD Graphics, HD and VGA dual display interfaces support 4K@60Hz.
- ◆Rich interfaces: 2 x1000M Realtek RTL8111H-LAN,2 xUSB3.0, 4 xUSB2.0, HDMI,VGA,AUDIO supports data storage and system boot.
Know which crawler you are allowing or blocking
A company can operate separate crawlers for search, model-training collection, and fetching a page at a user’s request. A rule for one crawler does not necessarily control the others. Use the provider’s current documentation to confirm names and purposes before editing your rules; crawler names, IP ranges, and policies can change.
| Provider | Search crawler | Training crawler | User-directed fetch |
|---|---|---|---|
| OpenAI | OAI-SearchBot, used to surface sites in ChatGPT Search |
GPTBot, which crawls content that may be used in model training |
ChatGPT-User, which may fetch a page after a user request |
| Anthropic | Claude-SearchBot |
ClaudeBot |
Claude-User |
Googlebot, used for Google Search |
Not identified as a training crawler in the provider distinctions described here | Not identified as a user-directed fetch agent in the provider distinctions described here |
OpenAI says the settings for OAI-SearchBot, GPTBot, and ChatGPT-User are independent. It recommends allowing OAI-SearchBot and requests from its published IP ranges for ChatGPT Search visibility. Its documentation says a robots.txt update may take about 24 hours to affect its search systems; this is provider guidance, not a guarantee that every crawler or cache updates on that schedule.
OpenAI describes ChatGPT-User as a user-initiated fetch rather than an automatic web crawler and notes that robots.txt rules may not apply to those actions. Anthropic distinguishes its search, training, and user-directed bots too, and says its bots respect robots.txt. Anthropic’s help documentation, dated April 7, 2026, says each subdomain you want to opt out must carry the directive. Treat user-directed fetches as a separate decision rather than assuming a training-crawler block will cover them.
Quick wins for a faster PC:
Repair Windows errors before they cause bigger problemsFix Now →Scan for outdated or missing drivers - takes under a minuteDriver Scan →Clear out junk files and repair common Windows errorsFree Scan →The distinction matters when a site wants to appear in an AI-powered search product but does not want to permit the same provider’s training crawler. Allowing a search crawler preserves its ability to fetch public pages for that search service; disallowing a separately named training crawler communicates a different preference. Neither choice guarantees that a page will be displayed, cited, ranked, or omitted. Those outcomes are determined by the provider’s systems.
Build purpose-specific robots.txt rules
For a public site that wants Google Search and ChatGPT Search visibility, but does not want the named OpenAI and Anthropic training crawlers to crawl its pages, a starting configuration could look like this:
User-agent: Googlebot
Allow: /
User-agent: OAI-SearchBot
Allow: /
User-agent: GPTBot
Disallow: /
User-agent: Claude-SearchBot
Allow: /
User-agent: ClaudeBot
Disallow: /
User-agent: *
Allow: /Publish the file at the site’s root, such as https://example.com/robots.txt, replacing the example domain with your own. The example explicitly allows Googlebot and the two named search crawlers, disallows the two named training crawlers site-wide, and leaves other crawlers allowed by default. A crawler must honor the instructions for these rules to affect its behavior.
This is not a universal recommended policy. Choose the search agents you want based on your visibility goals, and add user-directed fetch rules only after deciding whether you want those agents to retrieve public pages. For example, if you also want to ask compliant Anthropic user-directed fetches not to crawl the site, add a separate User-agent: Claude-User group with Disallow: /. The corresponding OpenAI group would name ChatGPT-User, but OpenAI says robots.txt may not apply to its user-initiated actions. Do not treat either setting as a confidentiality control.
Do these 3 things before closing this tab:
1Scan for outdated or missing drivers - takes under a minute2Repair Windows errors before they cause bigger problems3Fix the driver behind crashes, sound loss and screen glitchesRead the scope before you publish
Disallow: /asks a compliant crawler not to crawl any path on the host covered by that robots.txt file.- Rules can target paths rather than an entire site, but path-specific patterns should be checked against the crawler’s own documentation and tested before relying on them.
- A robots.txt file applies to the host and protocol where it is served. If your site uses multiple subdomains, publish and maintain the appropriate file for each one. Anthropic specifically says every subdomain being opted out needs its directive.
- Do not add a broad
User-agent: *disallow unless you intend to affect all crawlers matching that group. Keep provider-specific rules distinct when their purposes differ.
Anthropic’s documentation includes Crawl-delay: 1 as an example. Do not assume every crawler interprets that directive or that it offers an access restriction; check the relevant crawler documentation before using it.
Before publishing, compare the proposed file with the policy you wrote down. Check that you have not accidentally copied a broad disallow from an older configuration, omitted a search crawler you meant to permit, or put a rule only on the main domain when the relevant pages live on a subdomain. If you want a path-specific restriction, check the provider’s rule interpretation rather than assuming every crawler handles patterns identically. A simple test of the live file helps catch formatting or deployment mistakes, but it does not establish that every bot will obey it.
Rank #2
- 5 Gigabit Ethernet ports support high-speed LAN backbone infrastructures & gigabit WAN connections.
- With integrated SD-WAN, you can decrease you use of expensive MPLS or 4G/LTE connections and inspect traffic from home/small offices while improving resiliency and performance of your network.
- All logging and reporting functions included with purchase, with over 100 dashboards and reports including PCI and HIPAA.
Keep conventional search access intact
If you want Google Search visibility, do not block Googlebot as a shortcut for blocking AI training crawlers. Google says blocking Googlebot affects Google Search as well as Discover, Images, Video, and News. Separate crawler groups let you ask named AI training crawlers not to crawl while preserving Google’s access.
Google’s documentation also distinguishes crawling from indexing. If a page should remain crawlable but not appear in search, use a noindex directive instead of blocking Googlebot from the page. Google has to crawl the page to see a page-level noindex; if robots.txt prevents access, Google may not see the directive. For content that must not be accessible, use authentication or another access control.
What’s actually slowing this PC down?
Pick the symptom - the matching free tool is one click away.
Do not infer that keeping one search crawler allowed guarantees inclusion, ranking, or display in a search product. It preserves the crawler’s ability to access pages; search providers decide how their results are generated.
For supported files, Google documents that Googlebot crawls the first 2 MB, and for PDFs the first 64 MB. These are technical crawl limits, not a reason to block Googlebot or a substitute for deciding which pages should be available in search. If your question is whether a particular document is private, use access controls; if the concern is Google indexing, use an indexing directive that Google can access.
Check the CDN and WAF policy too
The origin’s robots.txt may not describe the complete policy visitors and bots encounter. A CDN or WAF can apply its own managed robots.txt behavior, bot detection, allow/block actions, or path-specific rules. Review the active settings at the delivery edge as well as the file served by your origin.
Cloudflare’s documentation groups bot behavior into Search, Agent, and Training classes and notes that one bot can have multiple behaviors. Its AI Crawl Control provides per-crawler allow or block choices, while more advanced exceptions can be implemented with WAF rules. The documentation describes user-agent detection on the free plan and says Bot Management detection IDs provide more thorough detection on upgraded plans. These are Cloudflare-specific capabilities; do not assume other CDN or WAF products use the same categories or controls.
Cloudflare documentation states that, from September 15, 2026, new domains have Training- and Agent-classified bots blocked on pages displaying ads while Search remains allowed; mixed Search/Training bots are also covered by training-blocking configurations. Because that date has passed, Cloudflare users should inspect the current effective zone policy rather than assume their origin file is the whole policy. The change is provider-specific and should not be generalized to other providers or every Cloudflare zone.
When checking the edge, look at the effective policy for the domain and paths you care about, not only whether a feature is enabled somewhere in the account. Consider whether a managed robots file is in use, whether a per-crawler action conflicts with the origin file, and whether a WAF exception applies to the paths you intended. Cloudflare documents both per-crawler controls and WAF rules for advanced exceptions; the right setup depends on the desired scope. A rule intended to preserve search access should be checked against the provider’s Search classification and the specific crawler, especially because one bot can have more than one behavior classification.
Verify the configuration and the crawler identity
- Write down the intended policy. List the search crawlers that should remain allowed, the training crawlers to disallow, and whether user-directed fetch agents should be allowed to retrieve public pages.
- Check the live robots.txt file. Open the root robots.txt for each relevant host or subdomain and confirm that the published file contains the groups you intended, with no older broad rule overriding your goal.
- Inspect the delivery layer. Review CDN-managed robots behavior, AI bot controls, and WAF rules. Check the effective settings for the relevant host and paths rather than inferring them from the origin file.
- Test the desired outcomes separately. Confirm that public pages remain crawlable by the search agents you chose, that training agents receive the intended policy where they honor robots.txt, and that pages requiring privacy are actually protected by authentication or another access control.
- Use logs and published verification methods. A request’s user-agent string can be spoofed. Google recommends reverse DNS checks or comparing a suspected Google crawler with its published IP ranges before treating it as Googlebot. Use each other provider’s published verification methods and your logs when assessing crawler traffic.
- Recheck after changes. If a provider changes its crawler names, IP ranges, or policy, update your inventory and test again. OpenAI says its search systems may take about 24 hours to reflect a robots.txt update, so an immediate result is not guaranteed.
Verification should match the goal. A successful fetch of robots.txt confirms what the host serves, but not that a noncompliant crawler will follow it. Seeing a URL absent from search results is a separate question from whether it is blocked from crawling. Likewise, receiving a crawler request in server logs does not by itself prove the identity claimed in its user-agent string. Treat these as separate checks rather than using one observation as proof of all the others.
Google specifically warns that user-agent headers are often spoofed. Before blocking a suspected Google crawler based on traffic, use Google’s recommended reverse DNS verification or compare the request with published Googlebot IP ranges. For other providers, use their published verification procedures where available. This reduces the risk of treating a request that merely claims to be a search crawler as an authentic search crawler.
Crashes, No Sound, or Screen Glitches?
Random freezes, missing sound and display glitches usually trace back to one bad driver. Find and replace yours safely.Free scan · under a minuteWindows Errors? Fix Them Before They Spread
Repair common Windows errors and clear accumulated junk for a smoother, more stable PC - no reinstall needed.Free scan · no reinstallCommon mistakes to avoid
Blocking every bot to stop training collection
A blanket block can also interfere with search crawlers you want to reach your public pages. Name the relevant crawlers separately and use a broad rule only when you intend to restrict all matching crawlers.
Assuming a robots.txt block hides a URL
Robots.txt is for crawl instructions, not guaranteed search-result removal. A blocked page may still be listed. For Google result exclusion, make the page crawlable so Google can see noindex; for private content, require authentication.
Assuming a crawler follows the file
Google explicitly warns that robots.txt cannot enforce crawler behavior: a crawler decides whether to obey it. A noncompliant crawler can ignore the file. Use server-side access controls when access itself must be denied.
Trusting a user-agent label on its own
User-agent strings are not proof of identity. Verify suspected Googlebot traffic using Google’s reverse DNS or published IP-range guidance, and consult provider-published verification methods for other crawlers before building a blocking rule around an apparent bot.
Changing the origin file but overlooking edge rules
A CDN or WAF can make a different decision from the origin’s robots.txt. When behavior does not match the file, inspect managed robots settings and bot or WAF rules at the delivery layer.
Treating user-directed fetches like training crawlers
A provider may use one agent to support search, another to collect material that may be used for training, and another to fetch a page after a person asks for it. Blocking the training crawler does not automatically decide whether a user-directed agent can fetch the same public page. Review those agents independently, and remember that OpenAI says robots.txt may not apply to ChatGPT-User’s user-initiated actions.
Choose controls by purpose, scope, and enforcement
A useful way to review a proposed policy is to ask four questions: what outcome do you want, which pages does it cover, which layer enforces it, and what happens if a crawler ignores the request?
| Control | Best fit | Scope | Important limitation |
|---|---|---|---|
Purpose-specific robots.txt group |
Request that a named, compliant crawler avoid crawling | Whole host or selected paths | Voluntary crawler preference; not a security boundary |
noindex |
Keep a crawlable page out of Google results | Page or resource carrying the directive | Google must be able to crawl the page to read it |
| Authentication or access control | Restrict content that should not be public | Protected resource or area | Requires a functioning access-control setup; robots.txt alone is not a substitute |
| CDN/WAF bot policy | Apply provider-specific bot actions or more advanced exceptions | Depends on the configured zone, crawler, and rule | Separate from origin robots.txt; behavior and plan capabilities vary by provider |
For many public sites, the practical combination is to leave search crawlers allowed, disallow the training crawlers whose collection you want to discourage, decide separately about user-directed fetches, and protect genuinely private material with access controls. Verify the resulting behavior at both the origin and delivery edge.
The Tool Desk
Outbyte Driver Updater FREEScan for outdated or missing drivers - takes under a minuteDriver Scan →Outbyte PC Repair FREERepair Windows errors before they cause bigger problemsFix Now →Rank #4
- Fortinet Web Application Firewall - virtual appliance for all supported platforms. Supports up to 1 x vCPU core
- Fortinet HW FWB-VM01
- Manufacturer Part: FWB-VM01
Frequently asked questions
Can I block GPTBot and still appear in ChatGPT Search?
OpenAI documents GPTBot for crawling content that may be used in model training and OAI-SearchBot for surfacing websites in ChatGPT Search. It says the settings are independent, so a site can disallow GPTBot while allowing OAI-SearchBot. Allowing the search crawler preserves access for that purpose but does not guarantee that a page will appear in search results. OpenAI recommends allowing OAI-SearchBot and requests from its published IP ranges for ChatGPT Search visibility.
Does blocking Googlebot block only Google’s AI features?
No. Google says Googlebot is the crawler for Google Search, and blocking it affects Google Search as well as Discover, Images, Video, and News. Do not use a Googlebot block as a substitute for naming a separate AI training crawler.
Will disallowing a URL in robots.txt remove it from Google?
Not necessarily. Google says a blocked URL may still appear in search results, including when Google learns about it from links. For Google result exclusion, use noindex and make sure Google can crawl the page to see the directive. For confidential content, require authentication or another access control.
Should I block ChatGPT-User or Claude-User?
That depends on whether you want user-directed agents to fetch public pages. They are distinct from the named training and search crawlers, so make the choice separately. Anthropic says its bots respect robots.txt and that each subdomain you want to opt out must have the directive. OpenAI says robots.txt may not apply to ChatGPT-User’s user-initiated actions, so its rule should not be treated as a guaranteed block.
How long does a robots.txt change take to take effect?
OpenAI’s current documentation says an update may take about 24 hours to affect its search systems. That is not a universal update schedule or guarantee for every crawler, provider, cache, or CDN. Check the relevant provider’s current guidance and avoid assuming an immediate change in observed behavior proves the rule is ineffective.
Is robots.txt enough to protect a private page?
No. Robots.txt is a preference mechanism for crawlers that choose to honor it, not an access boundary. Google warns that crawlers may ignore the instructions. Use authentication or another server-side control when a page must not be publicly accessible.
Does a CDN change the rules in my origin robots.txt?
A CDN or WAF may apply its own managed robots behavior, bot detection, allow/block decisions, or path-specific rules. Review the effective edge policy as well as the origin file. Cloudflare, for example, documents per-crawler AI Crawl Control settings and WAF rules for advanced exceptions; other providers may offer different features.
Do I need to pay for a product to block AI crawlers?
No specific physical product is necessary for the core setup described here: the central steps are crawler-specific robots.txt rules, suitable access controls for private content, and checking any CDN or WAF policy you already use. Cloudflare documents bot controls within its service, but whether to use them depends on your setup and the capabilities and plan available to your account. Do not treat a paid tool as a substitute for verifying the effective policy.
Frequently Asked Questions
Can I block GPTBot and still appear in ChatGPT Search?
OpenAI documents GPTBot for crawling content that may be used in model training and OAI-SearchBot for surfacing websites in ChatGPT Search. It says the settings are independent, so a site can disallow GPTBot while allowing OAI-SearchBot. Allowing the search crawler preserves access for that purpose but does not guarantee that a page will appear in search results. OpenAI recommends allowing OAI-SearchBot and requests from its published IP ranges for ChatGPT Search visibility.
Best Value
- Watchguard T145 Firebox with 1 Year Standard Support License (WGT145001) - The Firebox T145 delivers enterprise-grade protection for branch offices and retail sites. With a blend of 2.5Gb, 1Gb, and SFP/SFP+ ports, it supports high throughput, AI-driven malware protection, and DNS filtering for robust network defense.
- Standard Support covers software updates and round-the-clock emergency help. Add a Basic or Total Security Suite to activate IPS, gateway antivirus, and web filtering so threats are blocked before they reach users.
- Standard Support provides reliable technical assistance and software updates for WatchGuard Firebox appliances. Offering 24x7 help for emergencies and business-hours support for routine needs, it ensures your network stays secure and operational.
- Interfaces and deployment: 2.5Gb and 1Gb Ethernet with SFP or SFP+ fiber for clean aggregation and segmented backhaul at the edge.
- Performance and scale: UTM up to 710 Mbps with inspection on; flexible VPN topologies for hub and spoke or mesh designs.
Does blocking Googlebot block only Google’s AI features?
No. Google says Googlebot is the crawler for Google Search, and blocking it affects Google Search as well as Discover, Images, Video, and News. Do not use a Googlebot block as a substitute for naming a separate AI training crawler.
Will disallowing a URL in robots.txt remove it from Google?
Not necessarily. Google says a blocked URL may still appear in search results, including when Google learns about it from links. For Google result exclusion, use noindex and make sure Google can crawl the page to see the directive. For confidential content, require authentication or another access control.
Should I block ChatGPT-User or Claude-User?
That depends on whether you want user-directed agents to fetch public pages. They are distinct from the named training and search crawlers, so make the choice separately. Anthropic says its bots respect robots.txt and that each subdomain you want to opt out must have the directive. OpenAI says robots.txt may not apply to ChatGPT-User’s user-initiated actions, so its rule should not be treated as a guaranteed block.
Recommended Free Tools
How long does a robots.txt change take to take effect?
OpenAI’s current documentation says an update may take about 24 hours to affect its search systems. That is not a universal update schedule or guarantee for every crawler, provider, cache, or CDN. Check the relevant provider’s current guidance and avoid assuming an immediate change in observed behavior proves the rule is ineffective.
Is robots.txt enough to protect a private page?
No. Robots.txt is a preference mechanism for crawlers that choose to honor it, not an access boundary. Google warns that crawlers may ignore the instructions. Use authentication or another server-side control when a page must not be publicly accessible.
Does a CDN change the rules in my origin robots.txt?
A CDN or WAF may apply its own managed robots behavior, bot detection, allow/block decisions, or path-specific rules. Review the effective edge policy as well as the origin file. Cloudflare, for example, documents per-crawler AI Crawl Control settings and WAF rules for advanced exceptions; other providers may offer different features.
Do I need to pay for a product to block AI crawlers?
No specific physical product is necessary for the core setup described here: the central steps are crawler-specific robots.txt rules, suitable access controls for private content, and checking any CDN or WAF policy you already use. Cloudflare documents bot controls within its service, but whether to use them depends on your setup and the capabilities and plan available to your account. Do not treat a paid tool as a substitute for verifying the effective policy.
The Bottom Line
Keep crawler rules purpose-specific: allow Googlebot and any AI search crawlers you want to reach public pages, disallow the training crawlers you wish to discourage, and decide separately about user-directed fetches. Use noindex for Google result exclusion and authentication for private content. Then verify the live robots.txt file, crawler identity, and CDN/WAF policy; robots.txt is a request, not a security boundary.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
