iTechGuides is reader-supported. When you buy through links on our site, we may earn an affiliate commission. As an Amazon Associate I earn from qualifying purchases. Learn more
Antihost offers managed WordPress hosting with free migration, one-click staging and a 14-day free trial.
Yes—you can block named AI crawlers on WordPress without automatically blocking Google Search. Add separate User-agent groups to WordPress’s virtual /robots.txt, or enforce the same policy at your CDN or WAF. Block each identity separately: GPTBot, ClaudeBot, Google-Extended, and search or assistant agents such as OAI-SearchBot are different user agents. After every change, fetch the production file and check caches.
What blocking an AI crawler actually does
A WordPress site normally serves a virtual robots.txt response at its root. That file publishes your preferred crawling rules. A rule such as Disallow: / tells the named user agent not to request any path on the site:
The Tool Desk
Outbyte Driver Updater FREEScan for outdated or missing drivers - takes under a minuteDriver Scan →Outbyte PC Repair FREEClear out junk files and repair common Windows errorsFree Scan →User-agent: GPTBot
Disallow: /
This is a crawler preference, not authentication, encryption, or a network firewall. A compliant crawler reads the directive and stops making requests. A crawler that ignores it can still connect unless you add an enforcement layer such as a CDN/WAF rule, server access control, authentication, rate limiting, or another restriction.
Blocking one identity does not block every automated agent operated by the same company. For example, GPTBot, OAI-SearchBot, and ChatGPT-User are separate policy targets. Decide whether you want to stop model-training collection, search indexing, user-triggered retrieval, or all three.
#1 Best Overall
- Powerful Turbo Fan:CAGIWIRU Electric Air Duster have reached speed up to 130000RPM,Can efficient remove dust and debris.With LED light and Three Adjustable Speed to meet with different cleaning tasks.
- With Attached USB C Fasting Charging cable.Charing 3 hours,Enjoy up to 240 minutes of use on the lowest setting, For third speed can use for 25mins.with four charging options to meet with your needs.
- Wide Application: attached 5 different nozzles,making it suitable for a variety of scenes, such as car,pet, pc, keyboards, and other electronic devices. It also serves for office and home clean duster.
- Handy Design & Portable:Cordless air duster electric nozzle is Light weight and small size, design and comfortable to hold and space saving, convenient to carry around. You can put it in your room or inside the car without taking up space.
- Economical &Reusable: Built durable materials with a long-lasting battery,our CAGIWIRU air duster it's a sustainable alternative to disposable air cans.
Decide which traffic to restrict
Start with a written policy. The narrowest useful policy blocks training crawlers while leaving ordinary search engines and, if desired, search-oriented or user-requested retrieval agents alone. A broader policy blocks every named agent you do not want on the site.
| Goal | Typical user-agent targets | Effect |
|---|---|---|
| Restrict OpenAI model-training collection | GPTBot |
Signals that pages should not be collected by that training crawler. |
| Restrict OpenAI search or assistant retrieval | OAI-SearchBot, ChatGPT-User |
May reduce discovery or answers that depend on live retrieval. |
| Restrict Anthropic training collection | ClaudeBot |
Anthropic says a site-wide restriction signals that future materials should be excluded from its AI model-training datasets. |
| Restrict Google’s AI training use while considering Search separately | Google-Extended |
Targets Google’s extended AI-use crawler identity rather than ordinary Googlebot rules. |
| Restrict other named services | PerplexityBot, Bytespider, CCBot, Meta-ExternalAgent, Amazonbot, and others |
Each identity needs its own group or a maintained rule set. |
Cloudflare’s bot reference, updated April 23, 2026, distinguishes AI crawlers, AI search bots, AI assistants, and ordinary search engines. Use that distinction when deciding whether a training-only block is sufficient. Do not assume a vendor’s training bot, search bot, and assistant bot share one name.
Recommended Free Tools
Option 1: add rules to WordPress’s virtual robots.txt
Use manual rules when you have a short, deliberate list and want to keep normal search-engine directives unchanged. WordPress can generate a virtual file even when no physical robots.txt exists in the site root.
Before editing
- Record the current production response at
https://example.com/robots.txtor your own domain. - Copy every existing rule. A replacement that drops an existing
Allow, sitemap, or search-engine group can change crawling behavior unintentionally. - List the exact agents you intend to restrict. A spelling or capitalization error can make a rule ineffective.
- Decide whether the block applies to the whole site or only selected paths. The examples below use a site-wide block.
Site-wide training-crawler example
Place each crawler in its own group. Do not combine unrelated names when you need different decisions later:
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: Google-Extended
Disallow: /
These groups target three identities independently. They do not block ordinary Googlebot, and they do not automatically block OAI-SearchBot or ChatGPT-User.
Rank #2
- Screen Cleaner is Your Screen’s New Bestfriend! - EVEO is proud to present a screen cleaner that’s perfectly suited for your TV. Meet your new screen cleaner, computer screen cleaner, laptop cleaner, iPad cleaner, Macbook,tv cleaner or any other electronic device. Our screen cleaner spray and wipe along with the included plush microfiber cloth, easily removes dust, fingerprints & other smudges on the screen’s surface. Clean effectively for a streak-free screen
- Gentle on your Screens - Our screen cleaner spray and wipes is compatible to be used even for the most sensitive LCD, LED, CRT, and OLED screens in proper use. Be at ease knowing that this product is compatible with all the major brands: LG, SONY, Samsung, Sharp, iPad, iPhone, Android screens, Kindle, PC monitors - we’ve got you covered! Can be used as LCD screen cleaner, tv cleaner, smart tv screen cleaner, and more
- Premium Super Soft Plush Microfiber Cloth - Included in this efficient screen cleaning kit. This screen cleaner spray and wipe has our signature plush microfiber cloth specially designed to comply with screens and monitors and leave them streak-free. Has been shown to effectively absorbs stubborn stains, zero streaks, and no fiber shedding. Always use this together with our screen cleaner for the best results
- Easy-to-Use Screen Cleaner Kit + microfiber cleaning cloth - The EVEO Screen Cleaning Kit will easily be your new favorite tool when it comes to taking care of the things you love most, quick and easy solution for a streak-free screen. Simply screw on the sprayer gun on the screen cleaner bottle. Spray the cleaning liquid onto the microfiber cloth, then wipe the screen until desired cleanliness & shine is achieved.
- Quality Screen Care Kit Worth Purchasing - Enjoy a convenient Screen Cleaner anytime, anywhere you need it. Your satisfaction is our top priority. We stand behind the quality of our products and that’s why, you shouldn’t be worrying at all. We manufacture the Best Screen Cleaner. and can be used for laptop cleaning kit, laptop screen cleaner, tv screen cleaner for smart tv. MacBook screen cleaner, monitor cleaner or screen cleaner spray for electronic devices with microfiber cleaning cloth
Blocking additional identities
Add an explicit group for every agent covered by your policy:
Windows Errors? Fix Them Before They Spread
Repair common Windows errors and clear accumulated junk for a smoother, more stable PC - no reinstall needed.Free scan · no reinstallOutdated Drivers Are Slowing You Down
One free scan finds every outdated or missing driver and matches the right update for your exact hardware.Free scan · exact hardware matchUser-agent: OAI-SearchBot
Disallow: /
User-agent: ChatGPT-User
Disallow: /
User-agent: Claude-SearchBot
Disallow: /
User-agent: Claude-User
Disallow: /
User-agent: PerplexityBot
Disallow: /
Only add search and assistant identities if that is intentional. Blocking them can reduce retrieval-based visibility even when ordinary search indexing remains allowed.
Path-specific rules
If only a documentation, members, or draft area should be excluded, replace the root path with the path you need:
User-agent: GPTBot
Disallow: /private-docs/
Disallow: /drafts/
Path rules require careful testing. They are still advisory, and a URL that is linked or exposed elsewhere may remain discoverable even when its content is not fetched by a compliant crawler.
Keep existing search rules intact
Do not add a broad User-agent: * block merely to stop AI crawlers. That wildcard applies to every crawler that follows robots.txt, including search engines and other useful services. Preserve the site’s existing wildcard group unless you deliberately want a site-wide crawling change.
Do these 3 things before closing this tab:
1Fix the driver behind crashes, sound loss and screen glitches2Clear out junk files and repair common Windows errors3Scan for outdated or missing drivers - takes under a minuteRank #3
- 【Large size】：Measuring 800 x 300 mm/ 31. 5×12 inches, which is wider and taller than others. Large mouse pad can perfectly fit your keyboard and mouse of any size, and there is also ample space for peripherals such as phones, tablets, etc.
- 【LED backlight mouse pad】：Blade Hawks RGB mouse pad provides 7 static modes and 7 dynamic modes will give you the options you are looking for. Power-off memory function remembers the last lighting mode you selected. The super glow fiber Chroma will give you a colorful gaming setup that you want.
- 【Anti-slip rubber base】：Made from Natural rubber, this gaming mousepad will firmly grip your desktop, allowing you to enjoy the ultimate precision in the games you are most passionate about. It also comes with a texture that enhances this ability even further.
- 【Smooth and waterproof surface】: Micro-textured cloth surface, for extremely precise mouse control and a smooth movement. When liquid splashes on the gaming mouse pad, it forms water droplets and slides down. Will not delay your work or gameplay.
- 【Plug and play】: This mouse and keyboard pad is powered by USB, plug and play, no driver required. One button control light modes. Press one time to change the lighting mode, press twice to change the brightness, press and hold about 3 seconds turn ON/OFF.
Option 2: use a WordPress crawler-blocking plugin
A maintained plugin can provide a switchable list instead of requiring you to edit individual groups. The WordPress.org documentation for Block AI Crawlers says it can generate and maintain named rules for crawlers associated with OpenAI, Google, Meta, Anthropic, Perplexity, Apple, and Amazon.
Safe installation and configuration process
- Install the plugin from the WordPress administration area or another trusted distribution source, and confirm that it is maintained for your WordPress version.
- Open the plugin’s settings and review the complete crawler list before enabling anything. Separate training, search, and assistant identities where the interface allows it.
- Enable only the groups covered by your written policy. Leave ordinary search-engine rules unchanged unless you have a separate indexing decision.
- Save the settings, then request the site’s public
/robots.txtURL in a private browser window or with a command-line client. - Inspect the response for the exact
User-agentnames andDisallowvalues you selected.
The plugin documentation warns that a server, WordPress cache, or CDN may cache the generated file. A successful settings screen is not proof that visitors receive the new policy. Purge relevant caches and verify the live response.
Plugin failure modes
- Rules appear in the dashboard but not in the response: a page cache, reverse proxy, or CDN is serving an older file. Purge each layer and fetch the production URL again.
- A crawler remains unblocked: its current identity may not be in the plugin’s list, or it may be using a different search or assistant identity. Add the exact user agent or move enforcement to the edge.
- Search visibility changes unexpectedly: inspect the wildcard group and any rule that names a broad Google or other search user agent. Restore the previous group if it was overwritten.
- The plugin is abandoned or conflicts with another robots editor: disable one source of rules, retain a backup, and use a manually controlled or edge-managed policy.
Option 3: enforce the policy with a CDN or WAF
Use edge enforcement when compliance with robots.txt is not enough, when you need logs and rate controls, or when a bot ignores the file. Cloudflare documents an auto-updating “Block AI bots” managed rule and “Managed robots.txt,” which prepends AI-crawler directives. It also documents custom WAF rules and bot-management fields for blocking or challenging traffic at the edge.
| Approach | Scope | Enforcement | Maintenance | Main risk |
|---|---|---|---|---|
| Manual robots.txt | Specific named agents or paths | Advisory | You update names and policy | Ignored directives, stale caches, newly introduced names |
| WordPress plugin | Maintained list of named agents | Advisory through the generated file | Plugin may maintain signatures; you review settings | Plugin cache, conflicts, or incomplete list |
| CDN/WAF managed rule | Edge traffic matching vendor classifications | Blocks, challenges, or rate-limits requests | Vendor updates managed signatures; you audit impact | False positives, plan limits, and accidental search blocking |
| Custom server or WAF rule | Your chosen headers, paths, IP signals, or bot fields | Network or application enforcement | Fully yours to maintain | User-agent spoofing and brittle detection logic |
Configure edge controls conservatively
- Start in a logging, simulation, or challenge mode if your provider supports it.
- Match the documented bot identity or managed classification, not a casual substring such as “AI,” which can catch legitimate software.
- Exclude health checks, uptime monitors, authenticated integrations, and internal automation that your site needs.
- Review request logs for false positives and confirm that ordinary search-engine traffic is still permitted.
- Change to a hard block only after the observed traffic matches your policy.
- Keep a robots.txt rule as a machine-readable signal even when the WAF supplies the actual enforcement.
No edge rule can reliably identify every automated client if it spoofs a browser or user-agent string. Combine signals such as request behavior, provider bot fields, authentication, and rate limits, and monitor the result.
Free tools Windows power users keep installed
One-click scans. No signup required.
How to verify the live policy
Always test the production response rather than an editor preview. Replace the example host with your own domain:
Rank #4
- 【Softer and More Comfortable】Vaydeer wrist rest has unique diamond pattern, which is the combination of softness and aesthetics. The materials of wrist rest are improved into higher quality memory foam and covered with silky smooth lycra. The computer wrist rest makes you as comfortable and cushiony as like rest your wrists on clouds.
- 【Ergonomic Wrist Saver】The wrist rests for keyboard and mouse comes with a 17.32×3.15×0.83 inch keyboard wrist pad and a 5.94×3.15×0.83 inch mouse wrist support. Based on ergonomic design, the unique concave shape is the perfect fit for your wrist joints. The wrist rest pad fits most computer keyboards and laptops, improve hand and wrist posture, release your wrist and arm stress.
- 【Non-Slip Rubber Bottom】Featuring an anti-skid silicone base on the bottom, this wrist keyboard support stays firmly in place on your desk, preventing the padding from sliding around, ensuring stable and consistent wrist support during extended computer sessions.
- 【Better Experience & Pain Relief】Our keyboard arm rest is beneficial to alleviate the soreness caused by direct contact and friction between your arm and a hard desk surface, reducing the risk of wrist fatigue or carpal tunnel. The soft texture of memory foam can evenly distribute the pressure around your wrists and provide good support with just enough give.
- 【Helpful in Multiple Scenarios】Whether you're working, studying, writing, typing, gaming, this keyboard and mouse rest combo is an essential accessory to add comfort and support to your hands and wrists. It’s also a great gift for men, women, family, friend, coworker, gamer, teacher, etc.
curl -sSL https://example.com/robots.txt
Check that the response contains the intended groups and that an old cached copy is not being served. A useful verification pass is:
- Fetch the URL from a network outside your WordPress administration session.
- Search the response for every intended identity, including variants such as
GPTBot,OAI-SearchBot, andChatGPT-Userwhere applicable. - Confirm each group has the intended
Disallowpath and that no wildcard group was accidentally broadened. - Compare responses from the origin and CDN when you can access both. A mismatch indicates caching or edge configuration drift.
- Purge page, server, and CDN caches when the old file persists, then fetch again.
- Repeat the check after plugin updates, hosting migrations, DNS or CDN changes, and robots-editor changes.
Testing a user-agent string in a browser does not prove that a remote crawler will obey it. It confirms only what your server returns. For enforcement, inspect WAF logs and request outcomes as well.
What robots.txt cannot protect
- It is not access control. Do not put confidential material at a public URL and rely on
Disallow. Use authentication or remove the material from public delivery. - Compliance is voluntary. Cloudflare’s crawler analysis notes that some crawlers do not honor the file. Use a server, CDN, or WAF control for those clients.
- It does not erase previous collection. A new rule signals your future preference; it cannot recall copies already downloaded or datasets already assembled.
- Names can change. New services, search variants, and assistant identities require policy updates. A maintained list reduces, but does not eliminate, this work.
- User-agent strings can be spoofed. A rule that matches only a self-declared name is weak against a client that pretends to be another browser.
- Caching can delay changes. A WordPress plugin, host, reverse proxy, or CDN may serve an older virtual file until its cache expires or is purged.
Search visibility and referral trade-offs
Blocking AI traffic can reduce collection, but it may also reduce appearances in AI-generated answers or retrieval-based search features. Keep ordinary search rules separate from AI-specific groups so you can make that trade-off intentionally.
Cloudflare’s June 2025 analysis reported an OpenAI crawl-to-referral ratio of 1,700:1 and an Anthropic crawl-to-referral ratio of 73,000:1. Those figures are Cloudflare’s analysis for that period, not a universal rate for every site or a guarantee of traffic lost after blocking. They illustrate why publishers may decide that extensive crawling is not justified by observed referrals, while still recognizing that AI search and assistant visibility can have strategic value.
In the same June 2025 analysis, Cloudflare examined 3,816 top-domain robots.txt files and found 546—about 14% of that sample—with an allow or disallow directive aimed at AI bots. This is a result for Cloudflare’s selected sample and method, not a statistic about the entire web.
Best Value
- 【No Drill Mounted】Cable management tray can be assembled in as little as 3 minutes or less-simply. Side clips easily clampe the edge of desk, pass through the holes on the sides, and secure excess cables with baskets. Desk cable management supports inward or outward installation to meet your needs in different scenarios, and provides you with great convenience when collecting and organizing wires.
- 【Rubber Pad Protects Desk】Wire organizer under desk built-in soft rubber pads on the clamp protect your desk from scratches, so you will not damage your furniture or office.
- 【Sturdy and Beautiful】Cord management under desk is made of high-quality carbon steel and holds about 15 lbs. The wire basket has 2 holes on each side of the tray to enable your cables to pass freely. Cable baskets are suitable for desk thickness use from 0.4" to 2.4".
- 【Space Saving & Keep Organized】Desk cable management tray holds all your power cords, outlet strips, USB hubs and computer transformers hidden from your feet, and the data cables will not enter the vacuum cleaner to keep your desk clean and tidy. Desk wire organizer is also suitable for use in the kitchen and outdoors.
- 【Reduce Safety Risks】Cable tray under desk keeps all plugs away from your kids, no more worry about tripping. Using 2 holes wire tray saves extra space, helps you easily cabling and wire protection. The mesh design is not easy to accumulate dust, but avoids safety accidents caused by messy wires.
Troubleshooting by symptom
The rule is missing from /robots.txt
- Confirm the setting was saved and that the plugin or theme is the component generating the virtual file.
- Disable competing robots editors temporarily and retain a copy of the previous response.
- Purge WordPress, host, reverse-proxy, and CDN caches.
- Fetch the public URL with
curl -sSLrather than relying on an administrator preview.
The rule is present, but requests continue
- Check the exact user-agent name in access or WAF logs. The traffic may be using a search, assistant, or newly introduced identity.
- Remember that robots.txt is advisory. Move the policy to a WAF, server rule, authentication gate, or rate limiter when the client does not comply.
- Check whether the request is coming through a cached page or an alternate hostname that has a different robots file.
Google traffic dropped after the change
- Inspect for an unintended
User-agent: *or broad Google rule. - Keep
Google-Extendedin its own group if your intention was to target extended AI use rather than ordinary Google Search crawling. - Compare the current response with your saved pre-change copy and restore only the affected directive.
Legitimate automation is blocked by the WAF
- Review the matched rule, path, method, IP or provider field, and user-agent before adding an exception.
- Allow only the verified integration or path; do not disable all bot protection to fix one false positive.
- Retest from the integration’s real network and monitor logs after the exception.
A practical rollout checklist
- Inventory the AI crawler, search, and assistant identities relevant to your policy.
- Choose advisory robots.txt, a plugin-maintained list, edge enforcement, or a layered combination.
- Back up the current production robots response and existing search directives.
- Add separate groups for each blocked identity; avoid a broad wildcard block unless it is intentional.
- Publish the change and purge caches that can serve the virtual file.
- Fetch the production URL with
curl -sSLand inspect every group. - Review access and WAF logs to confirm the expected traffic behavior.
- Recheck after infrastructure, plugin, or provider changes, and whenever a new crawler identity appears.
Recommended policy patterns
Training-only restriction
Use separate blocks for training identities such as GPTBot, ClaudeBot, and Google-Extended, while leaving ordinary search and any deliberately permitted retrieval agents untouched. This is the least disruptive pattern for a site that still depends on search discovery.
All named AI access restricted
Block training, search, and assistant identities individually, then enforce the same decision with a CDN or WAF if non-compliance matters. This pattern is more comprehensive but can reduce AI-assisted discovery and requires more maintenance as names change.
Quick wins for a faster PC:
Repair Windows errors before they cause bigger problemsFix Now →Scan for outdated or missing drivers - takes under a minuteDriver Scan →Clear out junk files and repair common Windows errorsFree Scan →High-value or confidential areas
Do not rely on robots.txt. Put those areas behind authentication or another real access control, and use robots rules only as an additional signal for pages that remain public.
Bottom line
For most WordPress publishers, begin with explicit, separate robots.txt groups for the training crawlers you do not want, preserve ordinary search-engine rules, and verify the production response after cache purges. Add a maintained plugin when the list is large, and add CDN/WAF or server enforcement when a crawler ignores the advisory file. Treat GPTBot, ClaudeBot, Google-Extended, search bots, and assistant bots as distinct decisions rather than one universal “AI crawler” switch.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
