What’s actually slowing this PC down?
Pick the symptom - the matching free tool is one click away.
To protect a website from AI agents, combine crawler preferences in robots.txt with CDN or WAF rules, application-specific rate limits, and ongoing traffic monitoring. robots.txt tells compliant crawlers what you prefer; it does not technically prevent a client from requesting a page. Decide which automated traffic you want, then enforce those choices at the edge or in your application without blocking legitimate visitors.
Decide which automated traffic you want to allow
“AI bot” is not one traffic category. A site may want search indexing while declining model-training crawlers, or allow a real-time agent to retrieve public information while limiting its request rate. Make separate choices for each type of traffic rather than treating every automated request alike.
- Search crawlers: May be needed for pages to appear in search results.
- AI search or retrieval crawlers: May retrieve current pages for answers or other services.
- AI training crawlers: May collect content for model development.
- Real-time browser agents: Visit and interact with pages to complete user-directed tasks.
- Other automation: Uptime monitors and your own integrations may also need access.
Cloudflare describes bot categories based on behavior, including agent activity, rather than assuming every bot has the same purpose: Cloudflare’s bot concepts. AWS likewise discusses policies for AI crawlers and automated browser agents in its Bot Control use-case guidance.
Use robots.txt to state crawler preferences
Add rules to the site’s robots.txt file to communicate which paths a crawler should or should not fetch. The file is a policy signal, not an access-control mechanism: a crawler that does not follow the rules can still try to request those paths. Keep this distinction clear when protecting private material or managing server capacity; neither should depend on robots.txt alone.
The Tool Desk
Outbyte PC Repair FREERepair Windows errors before they cause bigger problemsFix Now →Outbyte Driver Updater FREEScan for outdated or missing drivers - takes under a minuteDriver Scan →#1 Best Overall
- The FortiGate 60F series offers an excellent Security and SD-WAN solution in a compact fanless desktop form factor for enterprise branch offices and mid-sized businesses
- Protect against cyber threats with industry-leading secure SD-WAN in a simple, affordable, and easy to deploy solution
- Security Identifies thousands of applications inside network traffic for deep inspection and granular policy enforcement Protects against malware, exploits, and malicious websites in both
- Provides Zero Touch Integration with Security Fabric's Single Pane of Glass Management Predefined compliance checklist analyzes the deployment and highlights the best practices to improve overall
A study evaluating seven named crawlers found that they respected robots.txt in the tested setup. That result is evidence about those crawlers under those study conditions, not a guarantee that all AI agents will comply. See the study, “Somesite I Used To Crawl: Awareness, Agency and Efficacy in Protecting Content Creators From AI Crawlers”.
Enforce access and capacity rules at the CDN or WAF
Review the controls provided by your CDN, hosting platform, or web application firewall (WAF). Depending on the service and configuration, these controls can monitor, block, rate-limit, or challenge bot traffic. AWS describes its WAF Bot Control feature for managing traffic such as scrapers, scanners, and crawlers.
Rank #2
- Entry-Level Privacy Gateway: Designed for users who want simple online privacy protection at an affordable level—ideal for basic home networking and daily internet use.
- Secure Browsing for Everyday Needs: Perfect for email, social media, online shopping, and standard streaming—protecting your connection while keeping setup and operation easy.
- Lightweight Protection Against Common Online Threats: Helps reduce exposure to unwanted ads, trackers, and risky websites, improving online safety for your household.
- Simple Setup, No Technical Skills Required: Plug it in, follow the quick steps, and start using—an excellent choice for beginners who don’t want complicated network configurations.
- Decentralized VPN (DPN) Included – No Monthly Payments: Get built-in decentralized VPN access with lifetime free usage, helping you stay private without paying recurring subscription fees
Where available, create distinct policies for the traffic categories you chose: for example, permit a desired search crawler, but challenge or limit an unwanted agent. AWS documents ways to allow selected AI bots while blocking or rate-limiting others. Cloudflare documents controls for AI bots by behavior in its Block AI Bots guide. Check the provider’s current interface and settings before deployment; product capabilities and defaults can change. Cloudflare notes that defaults for new domains changed on September 15, 2026.
Some services also offer challenges for automated browser sessions or mechanisms such as Web Bot Authentication for legitimate agents to identify themselves. These features can help distinguish traffic, but vendor documentation does not establish perfect classification. Test the policy against actual traffic and provide a way to identify false positives.
Free tools Windows power users keep installed
One-click scans. No signup required.
Rate-limit the routes that create risk
A single site-wide request threshold can be too blunt: a busy public page and an expensive search endpoint do not impose the same load. Set limits around routes and behaviors that are costly, easily enumerated, or prone to repeated access. Cloudflare’s rate-limiting best practices recommend tailoring rules to application use cases and explain that rate limits can work alongside bot management.
- Catalog or site search: Watch for repeated queries or rapid enumeration.
- Price or inventory lookups: Limit repeated requests that can be expensive or expose data at scale.
- Login and account routes: Apply controls appropriate to authentication traffic.
- APIs: Set limits that reflect the endpoint’s cost and the needs of legitimate integrations.
Choose thresholds from your application’s normal traffic and capacity, then observe the effect and adjust. The reviewed vendor guidance does not prescribe one universal limit that is right for every site. Pay particular attention to path matching and URL normalization: an edge rule and the origin application may interpret equivalent-looking paths differently.
Deploy controls without disrupting legitimate users
- Inventory automation. Identify desired search, AI retrieval, training, real-time agent, monitoring, and integration traffic. Record which categories should be allowed, limited, challenged, or blocked.
- Publish crawler preferences. Express the intended policy in
robots.txt, while treating it as guidance rather than enforcement. - Review existing edge rules. In your CDN, hosting, or WAF dashboard, check current bot policies and whether they allow, challenge, or block the traffic you intend to manage. AWS describes combining Bot Control with managed or custom rules in its use-case guidance.
- Add targeted rate limits. Start with the routes and request patterns that pose the greatest load or enumeration risk. Verify path matching and normalization behavior.
- Challenge selectively. Use verification where it addresses a specific risk, rather than adding friction indiscriminately across the site.
- Monitor and tune. Review request logs, origin load, response codes, and false positives after enabling rules. Test against real paths and traffic patterns, then adjust to preserve access for people and automation you want to serve.
AWS Prescriptive Guidance also describes static controls such as rate-based rules and bot activity signals: Static controls for managing bots.
Independent reader supportYour contribution helps us test, update, and keep practical guides available for everyone.Choose a managed option based on fit, not a universal ranking
If you are comparing AWS WAF Bot Control with Cloudflare controls, begin with your existing infrastructure and the signals and actions each option offers. Vendor documentation explains product capabilities, but it does not establish a head-to-head price comparison or independent efficacy ranking.
Recommended Free Tools
Best Value
- The FortiGate 60F series offers an excellent Security and SD-WAN solution in a compact fanless desktop form factor for enterprise branch offices and mid-sized businesses
- Protect against cyber threats with industry-leading secure SD-WAN in a simple, affordable, and easy to deploy solution
- Security Identifies thousands of applications inside network traffic for deep inspection and granular policy enforcement Protects against malware, exploits, and malicious websites in both
- Provides Zero Touch Integration with Security Fabric's Single Pane of Glass Management Predefined compliance checklist analyzes the deployment and highlights the best practices to improve overall
| What to compare | Why it matters |
|---|---|
| Where the site already runs | Existing CDN, cloud, and WAF setup affects implementation and operational fit. |
| Bot identity and behavior signals | These determine how the service categorizes traffic and which policies you can apply. |
| Available actions | Check support for rate limits, challenges, blocking, and custom rules. |
| Policy separation | Confirm whether you can treat search, training, and real-time agent activity differently. |
| Logging and tuning workflow | You need enough visibility to investigate blocked requests and correct false positives. |
| Cost for your traffic and required feature tier | Confirm current costs with the provider for your actual usage and needed capabilities. |
What layered protection can—and cannot—promise
Robots directives, WAF policies, challenges, and rate limits address different parts of the problem. Together with monitoring, they can reduce unwanted access and help keep request volume within the application’s capacity. The reviewed sources do not show that any single measure guarantees protection from every scraper or agent, nor do they establish a configuration that fits every site. Tune controls to your own routes and traffic, and check whether the rules are denying legitimate users or desired crawlers.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
