Free tools Windows power users keep installed
One-click scans. No signup required.
Cloudflare Pay Per Crawl and a custom HTTP 402 block response are separate features. Pay Per Crawl uses HTTP 402 to show pricing when a crawler has not supplied payment intent; a crawler that supplies the required intent can receive the content with HTTP 200. A custom 402 response, by contrast, is a message shown to a crawler that your rules block—it does not itself take payment or set up billing.
Cloudflare currently describes Pay Per Crawl as a closed beta, so first confirm that your account is eligible. The settings and steps below apply to Cloudflare’s documented dashboard controls as of October 4, 2026.
Which Cloudflare setting should you use?
| Goal | Cloudflare control | What it does |
|---|---|---|
| Charge for eligible crawler access | AI Crawl Control > Payments and Security | Sets a zone price and assigns crawlers the Charge action. A 402 can present the price when payment intent is absent; successful paid retrieval is HTTP 200. |
| Make selected paths free | Rules > Overview > Configuration Rule | Disables Pay Per Crawl for requests matching a URI pattern. |
| Set different prices by request or content | Dynamic pricing and an origin response header | Uses a crawler-price value instead of the zone’s default price when asking the crawler to pay. |
| Explain a block to a crawler | AI Crawl Control > Settings > Block response | Returns a configured 402 or 403 response for blocked crawlers; it is not the Pay Per Crawl payment flow. |
Cloudflare describes Pay Per Crawl as a feature for controlling and monetizing AI crawler access by setting a price per zone, and says it acts as Merchant of Record for the service. The controls above operate at different levels: crawler action, request path, price, or block response.
Check eligibility and expose the controls
Cloudflare lists Pay Per Crawl as a closed beta and directs interested site owners to its signup route or an Enterprise account executive. Beta access is not guaranteed.
#1 Best Overall
- VLAN Network Segregation: This router includes five preconfigured VLANs that isolate IoT devices, guest users, and work systems into separate, secure networks. Each LAN port and every WiFi SSID can be assigned to a VLAN, giving you complete control over how traffic flows inside your home.
- Dual VPN Client and Server Support: The router works as both a VPN client and a VPN server, supporting OpenVPN, IPsec, and WireGuard. You can route selected VLANs through a VPN while keeping others on your regular ISP connection, giving each device group the exact level of privacy it needs.
- Full WiFi 6 on Both Bands: With dual-band WiFi 6 support, the router delivers modern wireless performance across 2.4GHz b/g/n/ax and 5GHz a/n/ac/ax. It improves capacity, stability, and speed while remaining compatible with older devices, making it ideal for busy homes with many connections. Wi-Fi Mesh is available after firmware update.
- High-Performance Hardware Architecture: Powered by the IPQ6000 quad-core ARM processor at 1.2GHz, along with 128MB flash, 256MB RAM, and hardware NAT acceleration, the router handles multitasking, streaming, VPN traffic, and VLAN isolation smoothly without slowing your network.
- Flexible and Powerful Parental Controls: You can use trusted services like OpenDNS, CleanBrowsing, and Cloudflare for filtering, then add custom block lists, allow lists, and schedules. The router includes defenses against common bypass attempts, letting families create rules that match each user. Best of all, it's subscription free!
Before you configure a domain, make sure it is added to your Cloudflare account, uses Cloudflare nameservers or has DNS records managed by Cloudflare, and is visible to an Administrator or Super Administrator. To enable that visibility, open Manage Account > Settings > Pay Per Crawl and set the relevant domain to Visible. Cloudflare says visibility exposes configuration controls to domain administrators; it does not change the domain’s security rules.
Set the default price and choose crawler actions
- Open AI Crawl Control > Payments.
- Enable Pay Per Crawl, enter the default price per crawl, and save. Cloudflare’s 2026 pricing guide states a minimum of USD 0.001 per crawl. The configured price applies to each successful content retrieval (HTTP 200).
- Open AI Crawl Control > Security and select an action for each crawler.
| Action | Effect |
|---|---|
| Charge | Request payment at the configured price for successful access. |
| Allow | Allow access without charging. |
| Block | Deny access. |
There is one default price for crawlers assigned Charge; Cloudflare’s FAQ does not describe separate default prices per crawler. Cloudflare warns that charging or blocking search engine crawlers may keep them from indexing the site properly. Consider that impact before changing those crawler actions.
Make URI patterns free with a Configuration Rule
Use a Configuration Rule when you want Pay Per Crawl to apply generally but exclude particular paths. Cloudflare’s documented method matches the full URI with a wildcard and adds the Disable Pay Per Crawl setting.
- Go to Rules > Overview and select Create rule > Configuration Rule.
- Build the match using URI Full and the wildcard operator, then enter the URI pattern you want to exempt.
- Add Disable Pay Per Crawl as the setting and deploy the rule.
For example, a wildcard pattern can exempt a public directory or a free homepage. Cloudflare also lists these paths as always free to crawl: /robots.txt, /sitemap.xml, /security.txt, /.well-known/security.txt, and /crawlers.json.
Use dynamic pricing for content-specific rates
The zone price is the default. To return a different price for a request, enable dynamic pricing and have your origin return a crawler-price response header, such as crawler-price: USD 3.14. Cloudflare uses that value instead of the zone default. It forwards the header to the crawler only when the crawler is being asked to pay, in an HTTP 402 response.
Rank #2
- Lightning-fast Qualcomm Snapdragon SDX62 5G NR SA / NSA Modem Inside . The Cudy P5 supports 5G NR downlink speeds of up to 2.5 Gbps and 4G LTE downlink speeds of up to 1 Gbps. Wide spectrum bandwidth accelerates internet speed and reduces network latency for premium and time-sensitive mobile broadband services.
- Qualcomm IPQ5018 WiFi 6 SoC. 1 GHz Dual-core ARM Cortex-A53 CPU High Capacity 802.11ax SoC, delivers super fast dual band Wi-Fi with speeds of up to 2402 Mbps on the 5 GHz band and 574 Mbps on the 2.4 GHz band. Exceptional wireless performance enables online gaming and HD video streaming at the same time, while large files can be shared with multiple devices.
- Dual SIM and WAN Failover Keep You Always On-internet. Dual SIM slots provide redundancy and keep the device always online. Both SIM slots can be filled, you can choose whether to use SIM card 1 or SIM card 2, or auto select by Cudy. Set WAN/LAN port as WAN to enable Cudy use the landline internet from WAN, and 3G/4G connection works as a backup to provide a sustained and reliable internet connection for you.
- The replaceable cellular antenna interface provides a variety of installation possibilities. 4 x 5dBi cellular antenna and 2x5dBi WiFi antenna enhance the sensitivity of the router and improve the signal quality of 5G NR and Wi-Fi. At the same time, the cellular antenna is a detachable design. If you want to use an outdoor cellular antenna, the SMA connector also provides the possibility of an external cellular antenna.
- Multiple VPN Clients. With built-in PPTP/ L2TP / OpenVPN / WireGuard /IPsec/ Zerotier VPN, this 4G router can easily establish a connection to the VPN server to transport all your online data and traffic, securing it with its encryption at the same time. Compatible with 20 more DDNS providers, convenient to manage your remote cameras.
Cloudflare adds cf-pay-per-crawl to requests sent to the origin. Its documented values identify the active pricing mode:
zone-default: use the zone’s configured price.in-band: use a price supplied through the request’s pricing flow.bypass: bypass Pay Per Crawl for that request.
A Worker can set crawler-price based on the URL path, request properties, or content. Cloudflare says Snippets can inspect the cf-pay-per-crawl request header but cannot select crawler-price, because Snippets run too late in request processing. Cloudflare’s changelog dates the availability of URI exclusions and dynamic-pricing configuration to June 16, 2026.
Configure a custom 402 response for blocked crawlers
This setting applies to blocked requests, not to the Pay Per Crawl price negotiation. Cloudflare documents it for paid plans when at least one AI crawler is blocked.
Do these 3 things before closing this tab:
1Repair Windows errors before they cause bigger problems2Fix the driver behind crashes, sound loss and screen glitches3Clear out junk files and repair common Windows errors- Open AI Crawl Control > Settings > Block response.
- Choose 402 Payment Required or 403 Forbidden.
- Enter the response body as plain text and save the setting.
A custom 402 body can tell a blocked crawler to pay or contact the site owner, but the status and message do not establish payment processing. Cloudflare documents the actual per-crawl payment flow separately. If the associated AI Crawl Control WAF rule has been manually changed to return a status other than 402 or 403, the status-code control may be unavailable or blank.
Quick Recap
Understand rule precedence, billing, and failure cases
- WAF or Bot Management block: Cloudflare says these blocks override the Pay Per Crawl Charge action. A crawler stopped by either control does not proceed through the paid-crawl flow.
- Failed retrieval: Cloudflare’s FAQ says error responses are not billed. Successful recrawls are billed each time at the applicable configured price.
- Search visibility: Charging or blocking a search engine crawler can affect indexing; distinguish those crawlers from AI crawlers before applying a broad policy.
- Missing controls: Verify that the domain is visible in account settings and that your user has the required administrative visibility. Also confirm eligibility, since Pay Per Crawl is a closed beta.
- Unexpected 402: Determine whether the response came from Pay Per Crawl’s pricing flow or the configured block response. A 402 status by itself does not prove that a payment has been collected.
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
