Cloudflare’s AI Crawl Control is the main place to decide whether identified AI crawlers may access content, should be blocked, or can be charged through Pay Per Crawl. Payment configuration is in AI Crawl Control → Payments → Pay Per Crawl. A 402 response can mean either a block configured to return “Payment Required” or the first step in Pay Per Crawl’s signed payment exchange; those are different outcomes. WAF custom rules and bot settings can also stop a crawler before the payment flow runs.
Which settings decide whether a crawler can access your site?
In AI Crawl Control, review crawler activity and choose an action for each crawler:
- Allow: permits access under this crawler policy, but does not override other controls. WAF custom rules and robots.txt enforcement may still affect the request.
- Block: refuses access. The response can be configured as 403 Forbidden or 402 Payment Required, with a custom plain-text body.
- Charge: routes eligible crawlers through Pay Per Crawl when that feature is enabled. It is not simply an instruction to return a 402: the crawler must follow the payment handshake for a successful paid retrieval.
Cloudflare documents monitoring robots.txt compliance as part of AI Crawl Control. An Allow choice should therefore not be treated as a blanket bypass of robots.txt or the zone’s other security policies. See Cloudflare’s AI Crawl Control documentation and Pay Per Crawl pricing and configuration.
How do I enable bot payments with Pay Per Crawl?
- Open AI Crawl Control → Payments → Pay Per Crawl.
- Select Enable, set the default price, and save.
- Choose Charge for the crawlers that should be offered the paid-access flow.
Cloudflare labels Pay Per Crawl as a private beta; availability depends on account eligibility. The documented minimum is USD $0.001 per crawl. Cloudflare says this configured amount is charged for each successful content retrieval returning HTTP 200, not merely for sending a 402 challenge. Check the current account and documentation for current availability and settings.
Quick wins for a faster PC:
Repair Windows errors before they cause bigger problemsFix Now →Scan for outdated or missing drivers - takes under a minuteDriver Scan →Clear out junk files and repair common Windows errorsFree Scan →#1 Best Overall
- Read Before You Buy — No Video Output: These adapters support charging and USB 2.0 data transfer, but cannot transmit video signals. Except for standard USB webcams (which use USB data only), they are not compatible with HDMI/DisplayPort cables, video-capable USB-C hubs, or docking stations with video output.
- Convert USB-A Ports to USB-C: Designed to connect USB-C earphones, cables, flash drives, card readers, and other USB-C accessories to standard USB-A ports. Plug-and-play with no drivers or software required.
- Aluminum Alloy Housing: Built with a sturdy aluminum alloy shell that aids in heat dissipation and protects against daily wear and scratches. Designed to maintain a stable and secure connection.
- Compact & Travel-Friendly: The ultra-compact design allows the adapter to stay plugged into your device without blocking adjacent ports or adding bulk, reducing wear and tear on your original USB ports.
- 12-Month Warranty: Backed by a 12-month manufacturer warranty for peace of mind. Designed to meet strict quality control standards for reliable everyday performance.
What happens in the Pay Per Crawl 402 exchange?
- The crawler requests content set to Charge and receives HTTP/2 402 Payment Required with a
crawler-priceresponse header. - To retry, the crawler sends either
crawler-exact-pricematching the quoted amount orcrawler-max-pricecovering it. - The payment header must be included among the Web Bot Auth
signature-inputcomponents. If it is not signed as required, payment processing can fail. - After successful payment and content retrieval, the response is HTTP 200 and includes
crawler-charged. A failed paid request can return 402 withcrawler-error.
Cloudflare’s crawler-flow documentation describes this handshake. A 402 by itself does not confirm that a payment was collected. Cloudflare says repeat crawls incur the configured cost again, while /robots.txt, /sitemap.xml, /security.txt, /.well-known/security.txt, and /crawlers.json are always free.
How is a 402 block response different from a payment challenge?
A Block action can be configured to return either 403 or 402 and a custom plain-text message. That 402 communicates a refusal or payment/licensing condition; it is not, on its own, the Pay Per Crawl exchange described above. In the paid flow, the crawler receives a price header and must submit the appropriate signed payment header before successful access can return HTTP 200.
Rank #2
- 5-in-1 USB-C Hub: Experience comprehensive connectivity featuring a Power Delivery input, two USB-A 2.0 ports, a USB-A 3.0 port, and an HDMI port. (Note: The USB-C power delivery input port is only for connecting an external wall charger to power your laptop and cannot power peripheral devices.)
- 90W Pass-Through Charging: Achieve optimal charging with 90W pass-through power to your laptop, supported by a total input of 100W, with the hub reserving 10W for operational efficiency. (Note: Wall charger not included.)
- Quick Data Transfers: Accelerate your productivity with rapid data transfers using a high-speed 5Gbps USB 3.0 port and two 480Mbps USB 2.0 ports.
- 4K HDMI Display: Enhance your visual experience with a hub capable of delivering 4K resolution at 30Hz in both mirror and extend modes. Please note that this hub is compatible with MacBook (macOS 12 and newer), Windows 10 and 11, ChromeOS, and laptops equipped with DP Alt Mode and Power Delivery. Note: This device is not compatible with Linux.
- What You Get: Anker USB-C Hub (5-in-1, 4K HDMI), welcome guide, 18-month warranty, and our friendly customer service.
Cloudflare limits the AI Crawl Control response-code selector to 403 and 402. Its documentation warns that manually changing the associated WAF rule to a different response code can prevent AI Crawl Control from enforcing the selected code. See Cloudflare’s AI Crawl Control documentation.
Why might a crawler be blocked instead of charged?
Cloudflare documents this processing order: AI Crawl Control crawler-block WAF rules, then Cloudflare bot solutions, then Pay Per Crawl. A request blocked by an earlier control never reaches the charge step. In particular, Cloudflare says to turn off the Block AI Bots bot configuration when the intention is to use Pay Per Crawl. WAF custom rules can also affect crawlers marked Allow. See Cloudflare’s advanced AI Crawl Control guide and WAF integration guidance.
The Tool Desk
Outbyte PC Repair FREEClear out junk files and repair common Windows errorsFree Scan →Outbyte Driver Updater FREEScan for outdated or missing drivers - takes under a minuteDriver Scan →Rank #3
- Sleek 7-in-1 USB-C Hub: Features an HDMI port, two USB-A 3.0 ports, and a USB-C data port, each providing 5Gbps transfer speeds. It also includes a USB-C PD input port for charging up to 100W and dual SD and TF card slots, all in a compact design.
- Flawless 4K@60Hz Video with HDMI: Delivers exceptional clarity and smoothness with its 4K@60Hz HDMI port, making it ideal for high-definition presentations and entertainment. (Note: Only the HDMI port supports video projection; the USB-C port is for data transfer only.)
- Double Up on Efficiency: The two USB-A 3.0 ports and a USB-C port support a fast 5Gbps data rate, significantly boosting your transfer speeds and improving productivity.
- Fast and Reliable 85W Charging: Offers high-capacity, speedy charging for laptops up to 85W, so you spend less time tethered to an outlet and more time being productive.
- What You Get: Anker USB-C Hub (7-in-1), welcome guide, 18-month warranty, and our friendly customer service.
- Review WAF custom rules that match the crawler’s request.
- Check Cloudflare bot settings, including Block AI Bots, for an earlier denial.
- Confirm that the crawler has the intended Allow, Block, or Charge policy.
- Check robots.txt handling separately; an Allow action is not a universal override.
How can I exempt routes or set different prices?
Keep selected routes free
Create a Configuration Rule under Rules, use a URI Full condition for the routes to exempt (such as an exact homepage match or a public-directory wildcard), and enable Disable Pay Per Crawl. Cloudflare’s advanced guidance suggests leaving navigation and discovery pages available and excluding functional routes such as login, search, or APIs. The five paths listed above remain free regardless.
Use dynamic pricing
The zone price is the default. With dynamic pricing enabled, the origin can return a crawler-price response header—for example, crawler-price: USD 3.14—to replace the default for that request. Cloudflare says the price header is forwarded to the client when it is being asked to pay. This option is documented in the advanced configuration guide.
Rank #4
- Dual Converters, Infinite Potential：Includes 2× USB C male to USB A female adapters and 2× USB A male to USB C female adapters. Perfect for a wide range of uses—tablets with Bluetooth keyboards, expand USB ports on macbook, and more. Two different converters for all your daily needs
- Next-Level 10Gbps & 3A Charging: No more slow 480Mbps, this usb to usb c adapter has a transfer speed of up to 10Gbps, allowing you to do more transferring in less time. This usb adapter fits both USB A and USB C charger, supporting up to 3A fast charging
- Upgraded Exquisite Craftsmanship: With an aluminum alloy housing and metal connector, the usbc to usb adapter is extremely durable and sturdy. Rigorously tested to withstand more than 10,000 times of plugging and unplugging, ensuring long-lasting performance
- Broad Compatible: The usb c to usb adapter widely supports all USB C/ USB A devices like laptops, tablets, cellphones, car chargers, and phone chargers. Such as compatible with MacBook Pro/Air 2023/2022, Thunderbolt 4/3 Devices,Apple MagSafe Watch 9/8/7/SE/Ultra, iPad Pro 2022/2021, Samsung Galaxy S23/S20/S10, and iPhone 17/16/15 Pro. Plug and play
- Please Note: To reach 10Gbps speed, keep the cable under 3.3 ft. For USB A Male to USB C adapters, try flipping the USB C connector. USB C Male to USB A adapters support bidirectional 10Gbps transfer within 3.3 ft
When does the x402 Worker template make sense?
Cloudflare also offers an x402 Payment-Gated Proxy Worker template using the x402 protocol and HTTP 402. It is an implementation option for gating selected routes or charging bots while leaving human visits free, rather than another toggle for AI Crawl Control’s Pay Per Crawl. Use it when route-level payment logic is needed and you are prepared to deploy and configure a Worker-based flow. The template is covered in Cloudflare’s advanced AI Crawl Control documentation.
Quick Recap
Best Value
- 5-in-1 Connectivity: Equipped with a 4K HDMI port, a 5 Gbps USB-C data port, two 5 Gbps USB-A ports, and a USB C 100W PD-IN port. Note: The USB C 100W PD-IN port supports only charging and does not support data transfer devices such as headphones or speakers.
- Powerful Pass-Through Charging: Supports up to 85W pass-through charging so you can power up your laptop while you use the hub. Note: Pass-through charging requires a charger (not included). Note: To achieve full power for iPad, we recommend using a 45W wall charger.
- Transfer Files in Seconds: Move files to and from your laptop at speeds of up to 5 Gbps via the USB-C and USB-A data ports. Note: The USB C 5Gbps Data port does not support video output.
- HD Display: Connect to the HDMI port to stream or mirror content to an external monitor in resolutions of up to 4K@30Hz. Note: The USB-C ports do not support video output.
- What You Get: Anker 332 USB-C Hub (5-in-1), welcome guide, our worry-free 18-month warranty, and friendly customer service.
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
