iTechGuides is reader-supported. When you buy through links on our site, we may earn an affiliate commission. As an Amazon Associate I earn from qualifying purchases. Learn more
Do not trust a crawler’s User-Agent string as proof of identity: it is self-reported and can be spoofed. Check the request’s source IP against the operator’s published ranges or, for Google requests, use reverse DNS followed by a forward lookup. Then make a separate decision about whether the bot’s behavior and purpose are acceptable under your site policy.
What to check in a bot request
Keep the evidence needed to investigate a request: the source IP seen at your trusted edge or server, User-Agent, requested paths, timestamps, response codes, and request rate. If your site uses a CDN or reverse proxy, verify that the IP you check is the visitor IP supplied through trusted infrastructure. Do not trust an arbitrary forwarded-IP header: a client may be able to set one.
Treat the User-Agent as a label for what the requester claims to be, not as authentication. Google warns that crawler User-Agents can be spoofed and recommends verifying requests by DNS or published IP ranges. Google’s request-verification guidance is a concrete example of the difference between a claimed identity and a verified source.
Quick wins for a faster PC:
Scan for outdated or missing drivers - takes under a minuteDriver Scan →Clear out junk files and repair common Windows errorsFree Scan →Fix the driver behind crashes, sound loss and screen glitchesFind Drivers →How to verify a Google crawler or fetcher
For an individual request, Google’s documented manual check uses two-way DNS confirmation:
#1 Best Overall
- Compact and Efficient Design: The FortiGate 40F is designed for small to mid-sized businesses and enterprise branch offices, featuring a compact, fanless desktop form factor that ensures quiet operation and minimizes space usage.
- Robust Connectivity Options: Equipped with 5 GE RJ45 ports, including 1 WAN port and 4 internal ports, this model provides essential connectivity and flexibility for various network configurations in a small-scale environment.
- High-Performance Security: Offers up to 1 Gbps IPS throughput and 600 Mbps threat protection throughput, using Fortinet’s purpose-built security processor technology to deliver industry-leading performance and protection for SSL encrypted traffic.
- Advanced Threat Protection: Integrated with Fortinet’s AI-powered FortiGuard Labs, the FortiGate 40F offers comprehensive cybersecurity, identifying and mitigating both known and unknown threats to maintain robust security across your network.
- Simplified Management and Deployment: Features a user-friendly management console that provides comprehensive network automation and visibility, coupled with Zero Touch Integration with Fortinet’s Security Fabric for easy deployment.
- Take the source IP address from the request log and perform a reverse DNS lookup.
- Check that the resulting hostname ends in an accepted domain for the relevant fetcher class:
googlebot.com,google.com, orgoogleusercontent.com. - Forward-resolve that hostname.
- Confirm that the returned address includes the original source IP from your log.
A reverse lookup alone is insufficient: the forward lookup checks that the hostname resolves back to the address that connected. For recurring or high-volume checks, compare request IPs with the appropriate published Google crawler and fetcher IP ranges rather than running manual lookups for every request.
Identify which kind of Google request it is
Google publishes separate lists for common crawlers, special-case crawlers, and user-triggered fetchers. The distinction matters because their robots.txt behavior differs; a Google-branded request is not necessarily an ordinary Googlebot crawl.
Rank #2
- HARDWARE PLUS SECURITY SERVICES: FortiGate-60F Firewall Appliance bundled with 1 year of FortiCare Premium and FortiGuard Unified Threat Protection.
- UNIFIED THREAT PROTECTION (UTP): Secures against advanced online threats with comprehensive web filtering and anti-botnet technologies.
- OPTIMIZED FOR MEDIUM-SIZED BUSINESSES: Tailored for businesses needing robust security without the infrastructure of larger enterprises.
- RELIABLE CUSTOMER SUPPORT: FortiCare Premium ensures high-quality support and service continuity.
- EFFECTIVE PROTECTION: Employs advanced filtering technologies to safeguard against sophisticated threats.
- Common crawlers, such as Googlebot, follow robots.txt during automatic crawling.
- Special-case crawlers, such as AdsBot, may or may not follow robots.txt.
- User-triggered fetchers act because a user initiated a fetch and ignore robots.txt.
Use the corresponding class and range in Google’s crawler documentation when evaluating the request. Do not assume that a rule intended for ordinary crawling applies to every Google fetcher.
Do these 3 things before closing this tab:
1Clear out junk files and repair common Windows errors2Scan for outdated or missing drivers - takes under a minute3Repair Windows errors before they cause bigger problemsHow the verification methods compare
| Method | What it establishes | Coverage and best use |
|---|---|---|
| User-Agent string | The requester’s claimed identity; it does not prove who sent the request. | Useful as a clue for investigation, but not sufficient for an allowlist. |
| Reverse DNS plus forward confirmation | For Google requests, corroborates that the source IP maps to an accepted Google hostname and that hostname maps back to the IP. | Suitable for investigating an individual request; Google also recommends range matching for regular checks. |
| Published IP ranges | Whether the request IP matches a range published for a crawler or fetcher class. | More practical for automated, high-volume verification. Use the relevant operator’s official list. |
| Valid cryptographic signature | For a participating signed request, provides key-backed identity rather than relying only on self-declared headers and IP address. | Useful when present and correctly validated, but Google says its experimental rollout does not sign every request. |
What Web Bot Auth adds—and what it does not
Web Bot Auth uses cryptographic signatures to let participating bots prove a key-backed identity. Google’s guide describes its deployment as experimental and says it does not sign every request of a particular agent. Keep IP-based verification available rather than treating the signature method as universal. Google’s Web Bot Auth guide describes the protocol status and verification details.
Rank #3
- 【Up to 1100 Mbps VPN Speed 】 Hardware-accelerated WireGuard and OpenVPN-DCO deliver up to 1100 Mbps VPN throughput, over 3× faster than Brume 2 for smooth remote access and file transfers.
- 【Three 2.5G Ports & Multi-WAN】Tri-port 2.5GbE design with flexible WAN LAN configuration supports multi-gigabit wired setups, dual-ISP Multi-WAN and failover to keep home and SOHO networks online.
- 【Stealth VPN Obfuscation】VPN obfuscation disguises VPN traffic as regular HTTPS, helping you evade blocking, bypass restrictive networks and maintain stable, private connections.
- 【DPI protection】Deep Packet Inspection with visual dashboards blocks adult/gambling/malicious sites, while SQM and QoS prioritize gaming, calls, and video when bandwidth is tight
- 【OpenWrt & USB 3.0 Expansion】OpenWrt with 1GB DDR4 and 8GB eMMC lets you install plugins and build VPN, ad-blocking or NAS, while USB 3.0 Type‑C connects high-speed storage or 4G/5G dongles
For participating Google-Agent requests, the guide identifies a Signature-Agent header naming https://agent.bot.goog. The header by itself is not proof: validate the signature headers against the published key set and protocol. During the experimental rollout, Google recommends continuing to use IP addresses, reverse DNS, and User-Agent alongside signatures.
Decide whether to allow the traffic separately
Verification answers “who sent this request?” It does not answer “should this bot have access?” A real, authenticated crawler can still conflict with your preferences, make excessive requests, or require a different policy from search crawling.
Rank #4
- Runs UniFi Network for full-stack network management
- Manages 30+ UniFi Network devices and 300+ clients
- 1 Gbps routing with IDS/IPS
- Multi-WAN load balancing
- 0.96" LCM status display
Cloudflare’s Verified bots guidance, dated July 1, 2026, frames verification as both honest identification and non-abusive behavior. Its criteria include respecting crawl directives, using reasonable request rates, and not evading site-owner preferences or attacking sites. Its bot classifications include Search, Agent, Training, Data Collection, and Security Testing; a bot may have more than one behavior.
Use identity checks to inform a separate traffic decision:
- Allow a verified class of crawler when it serves a purpose you want and complies with your site rules.
- Rate-limit or monitor traffic that is authentic but too frequent or costly for your site.
- Block or investigate requests with an unverified identity, suspicious patterns, or behavior that violates your policy.
Applying the checks to AI crawlers beyond Google
There is no single documented verification recipe established here for every AI crawler operator. For a non-Google bot, look for that operator’s official documentation describing crawler identity, published IP ranges, signatures, or another verifiable mechanism. Do not infer that a familiar User-Agent is genuine, or apply Google’s DNS domains and ranges to another company’s traffic.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
