The Tool Desk
Outbyte PC Repair FREERepair Windows errors before they cause bigger problemsFix Now →Outbyte Driver Updater FREEScan for outdated or missing drivers - takes under a minuteDriver Scan →A WAF rule is better when you need to enforce a block on matching requests; robots.txt is better for telling compliant crawlers which content you prefer they not crawl. They solve different problems, so site owners often use both: publish clear crawler directives, then apply edge controls when request enforcement matters.
Does robots.txt block AI crawlers?
No. robots.txt communicates crawl preferences to bots that follow the Robots Exclusion Protocol; it does not grant or deny access. The IETF’s RFC 9309 is explicit: “These rules are not a form of access authorization.” A disallow directive is therefore not a way to protect confidential files or private pages. Use authentication or another access-control mechanism for that.
Under RFC 9309, crawlers use a product token to find matching user-agent groups. Matching is case-insensitive, matching groups are combined, and allow/disallow rules apply to URI paths; the most specific matching rule is used. These are protocol rules for crawlers that honor them, not a guarantee that a bot will comply.
How does a WAF rule differ?
A web application firewall evaluates incoming web or API requests and can take actions such as blocking or challenging traffic that matches configured conditions. Because the action is applied to matching requests before they reach the origin, a WAF is the stronger choice when the goal is enforcement. That enforcement is still bounded by what the provider can identify and what the rule actually matches: it is not a guarantee that every AI bot can be recognized.
Do these 3 things before closing this tab:
1Scan for outdated or missing drivers - takes under a minute2Repair Windows errors before they cause bigger problems3Fix the driver behind crashes, sound loss and screen glitches#1 Best Overall
- Read Before You Buy — No Video Output: These adapters support charging and USB 2.0 data transfer, but cannot transmit video signals. Except for standard USB webcams (which use USB data only), they are not compatible with HDMI/DisplayPort cables, video-capable USB-C hubs, or docking stations with video output.
- Convert USB-A Ports to USB-C: Designed to connect USB-C earphones, cables, flash drives, card readers, and other USB-C accessories to standard USB-A ports. Plug-and-play with no drivers or software required.
- Aluminum Alloy Housing: Built with a sturdy aluminum alloy shell that aids in heat dissipation and protects against daily wear and scratches. Designed to maintain a stable and secure connection.
- Compact & Travel-Friendly: The ultra-compact design allows the adapter to stay plugged into your device without blocking adjacent ports or adding bulk, reducing wear and tear on your original USB ports.
- 12-Month Warranty: Backed by a 12-month manufacturer warranty for peace of mind. Designed to meet strict quality control standards for reliable everyday performance.
| Question | robots.txt |
WAF rule |
|---|---|---|
| What it does | Publishes path directives for compliant crawlers to interpret. | Evaluates requests and can block or challenge matching traffic. |
| Does it enforce access? | No; it is not access control. | Can enforce request handling at the edge, subject to detection and configuration. |
| What can it target? | Crawler identity and URI path directives under the protocol. | Provider-dependent request conditions and actions, potentially with exceptions. |
| Main risk | A bot may ignore the published preference. | Rule conflicts, ordering, false identification, or unintended blocking. |
| Upkeep | Maintain the file and directives; managed features may update known crawler lists. | Custom rules may need manual updates; managed bot controls may update signatures. |
How to choose a policy for different AI bots
“AI crawler” is not one purpose. Search discovery, training, and agent or assistant activity can call for different decisions. A blanket block can stop traffic you wanted to allow, while an allow rule for one category should not be assumed to cover another.
Cloudflare’s bot reference, for example, labels GPTBot as an AI crawler, OAI-SearchBot as AI Search, and ChatGPT-User as an AI assistant. It also distinguishes ClaudeBot from Claude-SearchBot and lists PerplexityBot as AI Search, while Googlebot is labeled a search-engine bot. These are Cloudflare-specific labels and identifiers, not a universal or permanent taxonomy; Cloudflare points users to Radar for an up-to-date verified bot list.
Rank #2
- 5-in-1 USB-C Hub: Experience comprehensive connectivity featuring a Power Delivery input, two USB-A 2.0 ports, a USB-A 3.0 port, and an HDMI port. (Note: The USB-C power delivery input port is only for connecting an external wall charger to power your laptop and cannot power peripheral devices.)
- 90W Pass-Through Charging: Achieve optimal charging with 90W pass-through power to your laptop, supported by a total input of 100W, with the hub reserving 10W for operational efficiency. (Note: Wall charger not included.)
- Quick Data Transfers: Accelerate your productivity with rapid data transfers using a high-speed 5Gbps USB 3.0 port and two 480Mbps USB 2.0 ports.
- 4K HDMI Display: Enhance your visual experience with a hub capable of delivering 4K resolution at 30Hz in both mirror and extend modes. Please note that this hub is compatible with MacBook (macOS 12 and newer), Windows 10 and 11, ChromeOS, and laptops equipped with DP Alt Mode and Power Delivery. Note: This device is not compatible with Linux.
- What You Get: Anker USB-C Hub (5-in-1, 4K HDMI), welcome guide, 18-month warranty, and our friendly customer service.
- Want to express a crawl preference? Publish an appropriate
robots.txtdirective for the relevant crawler and path, and understand it depends on the crawler honoring the protocol. - Need to enforce a request policy? Use an edge rule that matches the traffic you intend to control, and check its interactions with other rules and exceptions.
- Need search crawlers to remain available? Separate search-related bots from training or agent traffic where your provider supports those distinctions; do not infer purpose from “AI” alone.
How to block AI bots without blocking search crawlers
- Decide which activity you want to allow. Define separate policies for search, training, and agent or assistant traffic rather than applying one blanket “AI” rule.
- Publish crawler preferences. Add path directives in
robots.txtusing the relevant crawler’s product token. Check that the rules are scoped to the intended paths and that more-specific allow/disallow matches produce the outcome you want. - Enforce where required. Configure a WAF or managed bot control for the traffic you intend to block. A robots directive alone does not enforce a block.
- Review rule order and exceptions. Confirm the block is not superseded or changed by other WAF, skip, redirect, transform, or allow rules. Recheck the result when managed bot settings or custom rules change.
- Verify the live configuration. Inspect the actual edge-provider settings and request handling for the site; do not assume a vendor’s announced defaults are active on your domain.
Cloudflare-specific controls and caveats
Cloudflare documents built-in bot controls and custom WAF rules as complementary options. Its managed settings include AI bot blocking and managed robots.txt; managed signatures are updated as Cloudflare identifies them, while custom rules require manual maintenance. The managed robots.txt feature can prepend Cloudflare-managed directives to an existing file. Its Content Signals categories for search, AI input, and AI training are vendor-documented preferences, not requirements of RFC 9309.
Rule precedence matters. Cloudflare says WAF rules are evaluated before AI Crawl Control’s pay-per-crawl feature, and upstream WAF rules can affect crawlers even when AI Crawl Control is configured to allow them. Skip, redirect, and transform rules may also interfere with an intended block. Review the order of custom rules and the AI Crawl Control rule, along with any exceptions. Cloudflare recommends custom rules rather than its dedicated User Agent Blocking feature for blocking specific user agents; that recommendation is specific to Cloudflare.
Free tools Windows power users keep installed
One-click scans. No signup required.
Rank #3
- Sleek 7-in-1 USB-C Hub: Features an HDMI port, two USB-A 3.0 ports, and a USB-C data port, each providing 5Gbps transfer speeds. It also includes a USB-C PD input port for charging up to 100W and dual SD and TF card slots, all in a compact design.
- Flawless 4K@60Hz Video with HDMI: Delivers exceptional clarity and smoothness with its 4K@60Hz HDMI port, making it ideal for high-definition presentations and entertainment. (Note: Only the HDMI port supports video projection; the USB-C port is for data transfer only.)
- Double Up on Efficiency: The two USB-A 3.0 ports and a USB-C port support a fast 5Gbps data rate, significantly boosting your transfer speeds and improving productivity.
- Fast and Reliable 85W Charging: Offers high-capacity, speedy charging for laptops up to 85W, so you spend less time tethered to an outlet and more time being productive.
- What You Get: Anker USB-C Hub (7-in-1), welcome guide, 18-month warranty, and our friendly customer service.
Cloudflare’s Block AI Bots page and changelog describe defaults scheduled for new domains on September 15, 2026: Training and Agent bots would be blocked on pages displaying ads, while Search would remain allowed, with mixed-purpose Search-and-Training bots included in training-block configurations. Because those pages describe a scheduled default rather than proving a particular domain’s current setup, verify the setting in your own dashboard before relying on it.
Independent reader supportYour contribution helps us test, update, and keep practical guides available for everyone.Which option should you use?
Use robots.txt to state crawl preferences for compliant bots. Use a WAF when you need edge enforcement for requests that match a correctly configured rule. For selective control, combine the two and distinguish bot purposes; then verify rule precedence and the site’s actual provider settings.
Quick Recap
Best Value
- 5-in-1 Connectivity: Equipped with a 4K HDMI port, a 5 Gbps USB-C data port, two 5 Gbps USB-A ports, and a USB C 100W PD-IN port. Note: The USB C 100W PD-IN port supports only charging and does not support data transfer devices such as headphones or speakers.
- Powerful Pass-Through Charging: Supports up to 85W pass-through charging so you can power up your laptop while you use the hub. Note: Pass-through charging requires a charger (not included). Note: To achieve full power for iPad, we recommend using a 45W wall charger.
- Transfer Files in Seconds: Move files to and from your laptop at speeds of up to 5 Gbps via the USB-C and USB-A data ports. Note: The USB C 5Gbps Data port does not support video output.
- HD Display: Connect to the HDMI port to stream or mirror content to an external monitor in resolutions of up to 4K@30Hz. Note: The USB-C ports do not support video output.
- What You Get: Anker 332 USB-C Hub (5-in-1), welcome guide, our worry-free 18-month warranty, and friendly customer service.
Rank #4
- Dual Converters, Infinite Potential：Includes 2× USB C male to USB A female adapters and 2× USB A male to USB C female adapters. Perfect for a wide range of uses—tablets with Bluetooth keyboards, expand USB ports on macbook, and more. Two different converters for all your daily needs
- Next-Level 10Gbps & 3A Charging: No more slow 480Mbps, this usb to usb c adapter has a transfer speed of up to 10Gbps, allowing you to do more transferring in less time. This usb adapter fits both USB A and USB C charger, supporting up to 3A fast charging
- Upgraded Exquisite Craftsmanship: With an aluminum alloy housing and metal connector, the usbc to usb adapter is extremely durable and sturdy. Rigorously tested to withstand more than 10,000 times of plugging and unplugging, ensuring long-lasting performance
- Broad Compatible: The usb c to usb adapter widely supports all USB C/ USB A devices like laptops, tablets, cellphones, car chargers, and phone chargers. Such as compatible with MacBook Pro/Air 2023/2022, Thunderbolt 4/3 Devices,Apple MagSafe Watch 9/8/7/SE/Ultra, iPad Pro 2022/2021, Samsung Galaxy S23/S20/S10, and iPhone 17/16/15 Pro. Plug and play
- Please Note: To reach 10Gbps speed, keep the cable under 3.3 ft. For USB A Male to USB C adapters, try flipping the USB C connector. USB C Male to USB A adapters support bidirectional 10Gbps transfer within 3.3 ft
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
