iTechGuides is reader-supported. When you buy through links on our site, we may earn an affiliate commission. As an Amazon Associate I earn from qualifying purchases. Learn more
Monitor automated traffic against a normal baseline, then investigate meaningful changes by endpoint, time, WAF rule or label, source, and response outcome. A bot label or traffic spike is a clue—not proof of malicious intent. Search crawlers, accessibility tools, uptime monitors, health checks, and partner integrations can all generate legitimate automated requests.
What to monitor
Use application, CDN, and web application firewall (WAF) logs or dashboards to establish typical request patterns. If traffic data is spread across multiple systems, correlate it centrally; AWS recommends centralized logging when multiple sources are involved.
- Rates over time: Compare requests with the usual level and with total site traffic. A changing rate or share is often more informative than a raw count.
- Paths and endpoints: Track which pages and APIs receive requests, especially login, account creation, checkout, and other high-impact functions.
- Rules and labels: Watch which WAF rules match and which labels appear most often. AWS advises tracking top-hit rules and labels; a change can reflect targeted activity or a false positive.
- Sources and request patterns: Review source distribution, user-agent patterns, and geographic concentration where available. These are investigative clues, not identity guarantees.
- Outcomes: Compare requests with application responses, such as login failures or other errors, to understand what the traffic did rather than relying on request volume alone.
AWS documents traffic overviews, detailed WAF logs, labels, CloudWatch metrics, and anomaly detection. Its guidance on monitoring WAF activity is at AWS WAF application-layer DDoS best practices.
How to recognize activity worth investigating
Look for deviations from your baseline, then check whether independent signals point to the same endpoint or behavior. A sudden rise in one rule label, for example, matters more if it coincides with concentrated requests to login or an API and a change in application outcomes. A spike alone may have a benign cause, including a newly active monitor or a rule that started matching legitimate users.
#1 Best Overall
- Investigate unusual increases in request rate or in the share of total traffic.
- Check whether requests cluster around sensitive paths rather than being evenly spread across the site.
- Compare source, user-agent, rule or label, and response patterns; do not treat any one as conclusive.
- Review the activity in application context before changing a rule or blocking traffic.
Cloudflare’s bot guidance identifies automated-traffic volume, targeted pages, score patterns, unusual user-agent volume, and geographic concentration as signals to review. These signals help direct an investigation; they do not establish intent by themselves. See Cloudflare bot analytics.
Give sensitive endpoints closer attention
Site-wide limits can miss concentrated activity against a particular function or disrupt unrelated users. Monitor login, account creation, checkout, and APIs in their application context, and set narrower thresholds for higher-risk paths where appropriate. AWS recommends rate-based rules for sensitive URIs such as login and account creation; Cloudflare documents path-specific rate-limit examples and event review.
There is no universal request threshold that proves abuse. A workable limit depends on the endpoint, normal user behavior, and your service configuration. Treat provider examples as configuration examples, not general recommendations, and inspect events to see whether a threshold would affect legitimate users. Cloudflare’s documentation is available at Rate limiting rules.
Keep legitimate automation working
“Automated” describes how requests are made, not whether they are harmful. Identify expected search crawlers, accessibility tools, internal uptime monitors, health checks, and partner integrations before enforcement. AWS notes that health checks and uptime monitors may be identified as bots; Cloudflare advises allowing verified bots and expected internal tools.
Use multiple layers of evidence. Static checks and rate limits can help with simpler or high-volume patterns, while behavioral or client signals may add context. OWASP describes controls across edge, application, and backend or business layers, and emphasizes that the goal is not to block every bot but to make abusive automation more costly without disrupting legitimate users and bots. See the OWASP Bot Management and Anti-Automation Cheat Sheet.
Review and tune before enforcement
- Observe first. Establish normal rates, paths, sources, and outcomes using your application, CDN, and WAF data.
- Inspect what a rule would affect. Where available, use a count or observation mode before switching a rule to blocking. AWS specifically recommends count mode first.
- Check for false positives. Review security events and confirm whether legitimate crawlers, monitors, health checks, or users would be affected. Add appropriate exceptions for expected traffic.
- Choose a proportionate action. Depending on the evidence and risk, consider a challenge or step-up verification before a hard block. AWS also describes forwarding suspicious labels to an application for additional verification.
- Keep reviewing after changes. Monitor events and outcomes after deployment, and tune rules if legitimate activity is caught or the suspicious pattern changes.
Cloudflare’s guidance covers reviewing bot analytics and security events and tuning controls to reduce false positives: Bot analytics and WAF managed rules.
Independent reader supportYour contribution helps us test, update, and keep practical guides available for everyone.Tools and service-specific differences
Start with the logs and dashboards already available in your application, CDN, and WAF. When assessing a monitoring setup, check whether it provides useful path-level and request-level visibility, log export and correlation, endpoint-specific thresholds, verified-bot handling, an observation mode, and gradual enforcement options. Fit with your current stack matters; no single provider or signal is universally best.
Recommended Free Tools
| Option | Documented monitoring capabilities | Availability qualification |
|---|---|---|
| AWS WAF and CloudWatch | Traffic overviews, WAF logs, labels, CloudWatch metrics, and anomaly detection; AWS recommends examining rule and label patterns in context. | Capabilities are described in AWS documentation; availability depends on the services and configuration in use. |
| Cloudflare bot analytics and security controls | Bot analytics, targeted-path views, security events, rate limits, and rule review. | Cloudflare says full bot analytics require Business or above, while basic security metrics are available to Free and Pro users. Plan features can change. |
These are examples of documented capabilities, not independent performance comparisons. Cloudflare’s bot analytics documentation is at Cloudflare Bot Analytics.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
