What’s actually slowing this PC down?

Pick the symptom - the matching free tool is one click away.

Special offer. See more information about Outbyte and uninstall instructions. Please review EULA and Privacy policy.

iTechGuides is reader-supported. When you buy through links on our site, we may earn an affiliate commission. As an Amazon Associate I earn from qualifying purchases. Learn more

AI crawlers slow a WordPress site when their requests bypass the cache and execute PHP, plugins, theme code, database queries, searches, logins or APIs. The durable fix is not a blanket user-agent block: measure which bots and paths consume resources, cache what can be cached, and enforce selective challenges or blocks at the CDN/WAF edge before unwanted traffic reaches the origin.

Why AI crawlers can become a WordPress hosting problem

A request for a cached article can be inexpensive. A request that misses the page cache may start PHP workers, load the active theme and plugins, query MySQL, generate an archive or search result, and send the response over the network. Hundreds of concurrent crawler requests multiply that work. The resulting symptoms are slow time to first byte, exhausted PHP workers, high CPU or memory use, database-connection saturation, bandwidth spikes, HTTP 5xx responses and hosting-account resource-limit notices.

Special offer. See more information about Outbyte and uninstall instructions. Please review EULA and Privacy policy.

The bot itself is not proof of abuse. A crawler that fetches a small number of cacheable pages may have little effect, while a modest-looking bot that repeatedly requests uncached searches or API endpoints can be expensive. Measure the mechanism in your logs and metrics instead of assuming every automated request is harmful.

How large is AI-bot traffic?

Cloudflare’s 2025 Radar analysis found AI bots generated an average of 4.2% of HTML requests. The monthly range was 2.4% to 6.4%. Googlebot averaged 4.5% and reached 11% in late April. Those figures describe Cloudflare’s observed traffic, not a forecast for every WordPress site, but they show why a single percentage of requests can matter when those requests concentrate on uncached or computationally expensive URLs.

#1 Best Overall
Sale
Compressed Air Duster,130000RPM Super Power Cordless Air Duster,Rechargeable Brushless Blower,3 Gear Adjustable Blower with LED Light,Dust Remover for Computer/House/Outdoor/Car
  • Powerful Turbo Fan:CAGIWIRU Electric Air Duster have reached speed up to 130000RPM,Can efficient remove dust and debris.With LED light and Three Adjustable Speed to meet with different cleaning tasks.
  • With Attached USB C Fasting Charging cable.Charing 3 hours,Enjoy up to 240 minutes of use on the lowest setting, For third speed can use for 25mins.with four charging options to meet with your needs.
  • Wide Application: attached 5 different nozzles,making it suitable for a variety of scenes, such as car,pet, pc, keyboards, and other electronic devices. It also serves for office and home clean duster.
  • Handy Design & Portable:Cordless air duster electric nozzle is Light weight and small size, design and comfortable to hold and space saving, convenient to carry around. You can put it in your room or inside the car without taking up space.
  • Economical &Reusable: Built durable materials with a long-lasting battery,our CAGIWIRU air duster it's a sustainable alternative to disposable air cans.

Which WordPress paths cost the most?

  • Uncached home, category, tag and date-archive pages that run the full WordPress stack.
  • Internal search, especially queries with many results or no cache coverage.
  • /wp-login.php and other authentication endpoints.
  • /xmlrpc.php, feeds and REST API routes that are repeatedly polled.
  • WooCommerce cart, checkout and account paths, which are normally personalized and uncacheable.
  • Custom plugin endpoints, webhooks and expensive reporting or filter URLs.

Review response time, cache status and database activity for each path. A bot hitting a cached article may be less urgent than a bot repeatedly missing cache on a search or checkout URL.

Diagnose the traffic before blocking it

Make a baseline and keep a rollback plan before changing rules. A sudden reduction in bot requests is not a success if it also blocks search indexing, uptime monitoring, customer logins or a legitimate integration.

Special offer. See more information about Outbyte and uninstall instructions. Please review EULA and Privacy policy.
  1. Correlate resource metrics. Compare origin CPU, memory, PHP-worker utilization, database connections, bandwidth and response time with request-rate spikes. Note whether the slowdown starts at the edge or only when traffic reaches the origin.
  2. Segment access logs. Group requests by user-agent, source IP, autonomous-system number (ASN), URL path, method, status code and cache hit or miss. Record request counts and concurrency, not just the number of unique bots.
  3. Classify the automation. Separate verified search crawlers, AI training crawlers, AI-search or user-action crawlers, uptime monitors, authenticated integrations and clearly abusive automation. Treat an unverified user-agent string as a clue, not identity proof.
  4. Find the expensive combinations. Look for a bot that produces cache misses, invokes PHP, returns 4xx or 5xx errors, or requests login, search, XML-RPC, REST, feed or commerce paths at high rates.
  5. Record a before state. Save origin load, cache-hit ratio, top paths, bot volume, 4xx/5xx rates and search-console crawl signals. Keep the existing rule configuration so you can undo one change at a time.

What to look for in logs

A user-agent label alone is weak evidence because it can be forged. Combine it with IP reputation, ASN, request behavior, headers, verification results and whether the source follows published crawl preferences. A high request rate from many addresses, repeated query variations and no meaningful cache reuse are stronger indicators of a problem than the name “GPTBot” or “ClaudeBot” by itself.

Reduce origin work with caching and edge enforcement

Cache the responses that do not need PHP

Full-page caching lets a reverse proxy or CDN serve a public article without starting WordPress for every request. Object caching can reduce repeated database queries for requests that still need PHP. Review exclusions carefully: logged-in sessions, carts, checkout, account pages and other personalized responses must not be served from a public cache.

Rank #2
EVEO® Screen Cleaner Spray Kit (Streak-Free) TV Screen Cleaner for Smart TV, Lcd, LEd, Oled | Computer Screen Cleaner Spray and Wipe Microfiber Cloth - TV, Monitor, Laptop, iPad, Car - 6.7oz 1 Pack
  • Screen Cleaner is Your Screen’s New Bestfriend! - EVEO is proud to present a screen cleaner that’s perfectly suited for your TV. Meet your new screen cleaner, computer screen cleaner, laptop cleaner, iPad cleaner, Macbook,tv cleaner or any other electronic device. Our screen cleaner spray and wipe along with the included plush microfiber cloth, easily removes dust, fingerprints & other smudges on the screen’s surface. Clean effectively for a streak-free screen
  • Gentle on your Screens - Our screen cleaner spray and wipes is compatible to be used even for the most sensitive LCD, LED, CRT, and OLED screens in proper use. Be at ease knowing that this product is compatible with all the major brands: LG, SONY, Samsung, Sharp, iPad, iPhone, Android screens, Kindle, PC monitors - we’ve got you covered! Can be used as LCD screen cleaner, tv cleaner, smart tv screen cleaner, and more
  • Premium Super Soft Plush Microfiber Cloth - Included in this efficient screen cleaning kit. This screen cleaner spray and wipe has our signature plush microfiber cloth specially designed to comply with screens and monitors and leave them streak-free. Has been shown to effectively absorbs stubborn stains, zero streaks, and no fiber shedding. Always use this together with our screen cleaner for the best results
  • Easy-to-Use Screen Cleaner Kit + microfiber cleaning cloth - The EVEO Screen Cleaning Kit will easily be your new favorite tool when it comes to taking care of the things you love most, quick and easy solution for a streak-free screen. Simply screw on the sprayer gun on the screen cleaner bottle. Spray the cleaning liquid onto the microfiber cloth, then wipe the screen until desired cleanliness & shine is achieved.
  • Quality Screen Care Kit Worth Purchasing - Enjoy a convenient Screen Cleaner anytime, anywhere you need it. Your satisfaction is our top priority. We stand behind the quality of our products and that’s why, you shouldn’t be worrying at all. We manufacture the Best Screen Cleaner. and can be used for laptop cleaning kit, laptop screen cleaner, tv screen cleaner for smart tv. MacBook screen cleaner, monitor cleaner or screen cleaner spray for electronic devices with microfiber cleaning cloth

Set appropriate cache-control rules, purge content when it changes, and verify that query strings, cookies and authorization headers are not accidentally turning every request into a miss. Measure the hit ratio for the paths bots actually request rather than relying on a site-wide average.

Filter at the edge, not after WordPress starts

A CDN or reverse proxy can challenge or reject a request before it consumes origin CPU. Where your architecture supports it, restrict the origin firewall to the proxy’s published IP ranges so attackers cannot bypass edge controls by connecting directly to the server. Confirm that health checks, deployment tools and trusted integrations can still reach the origin through the permitted path.

Special offer. See more information about Outbyte and uninstall instructions. Please review EULA and Privacy policy.

Edge controls have lower origin cost and usually better burst handling than rules implemented in a WordPress plugin. They also need observability and a rollback method: inspect sampled requests, rule matches and challenge outcomes after each change.

Allow useful crawlers and challenge the rest

Do not treat all automation as interchangeable. Verified Googlebot and Bingbot traffic can support search visibility, while an uptime monitor may be required for incident detection. Preserve those functions explicitly and apply stricter controls to unverified or unwanted AI crawlers.

Rank #3
Sale
BladeHawks RGB Gaming Mouse Pad Extra Large Extended 31.5x12 Inch-Black | LED mouse pad with 14 lighting modes, covers keyboard & mouse, anti-slip rubber base, waterproof surface, USB powered
  • 【Large size】:Measuring 800 x 300 mm/ 31. 5×12 inches, which is wider and taller than others. Large mouse pad can perfectly fit your keyboard and mouse of any size, and there is also ample space for peripherals such as phones, tablets, etc.
  • 【LED backlight mouse pad】:Blade Hawks RGB mouse pad provides 7 static modes and 7 dynamic modes will give you the options you are looking for. Power-off memory function remembers the last lighting mode you selected. The super glow fiber Chroma will give you a colorful gaming setup that you want.
  • 【Anti-slip rubber base】:Made from Natural rubber, this gaming mousepad will firmly grip your desktop, allowing you to enjoy the ultimate precision in the games you are most passionate about. It also comes with a texture that enhances this ability even further.
  • 【Smooth and waterproof surface】: Micro-textured cloth surface, for extremely precise mouse control and a smooth movement. When liquid splashes on the gaming mouse pad, it forms water droplets and slides down. Will not delay your work or gameplay.
  • 【Plug and play】: This mouse and keyboard pad is powered by USB, plug and play, no driver required. One button control light modes. Press one time to change the lighting mode, press twice to change the brightness, press and hold about 3 seconds turn ON/OFF.

Use bot scores as one signal

Cloudflare documents a bot score from 1 to 99. Lower scores indicate a greater likelihood of a script or automated agent; higher scores are more consistent with normal browsers. Its documented examples allow verified bots, block score-1 traffic and apply a managed challenge to scores 2 through 29. Cloudflare describes the score as a likelihood that a request originates from a bot, not as proof of a particular organization.

Use a staged policy: observe scores first, challenge suspicious traffic on low-risk public paths, and block only when the logs show that the rule is catching unwanted requests. Exempt authenticated APIs, monitoring systems and verified crawlers explicitly. A challenge on a machine-to-machine API can break an integration even when it is not malicious.

Special offer. See more information about Outbyte and uninstall instructions. Please review EULA and Privacy policy.

Apply stricter rules to expensive paths

Protect /wp-login.php, /xmlrpc.php, internal search and expensive API routes more aggressively than cacheable public articles. A path-specific rule can require a challenge for suspicious requests, impose a lower rate threshold, or deny traffic from a confirmed abusive source. Keep normal public browsing and verified search crawling on a less restrictive policy.

Rate-limit bursts without expecting mathematical precision

A Cloudflare rate rule consists of a matching expression, counting characteristics, a period, a request threshold, an action and a mitigation duration. Start with a narrow expression for the path and method that your logs identify as abusive. Count by a meaningful characteristic, such as source IP, and review whether a shared corporate or mobile address could represent many legitimate users.

Rank #4
Sale
Vaydeer Wrist Rest for Keyboard and Mouse, Computer Ergonomic Wrist Support Pad, Soft Memory Foam Arm Cushion for Desk, Palm Hand Office Laptop Typing
  • 【Softer and More Comfortable】Vaydeer wrist rest has unique diamond pattern, which is the combination of softness and aesthetics. The materials of wrist rest are improved into higher quality memory foam and covered with silky smooth lycra. The computer wrist rest makes you as comfortable and cushiony as like rest your wrists on clouds.
  • 【Ergonomic Wrist Saver】The wrist rests for keyboard and mouse comes with a 17.32×3.15×0.83 inch keyboard wrist pad and a 5.94×3.15×0.83 inch mouse wrist support. Based on ergonomic design, the unique concave shape is the perfect fit for your wrist joints. The wrist rest pad fits most computer keyboards and laptops, improve hand and wrist posture, release your wrist and arm stress.
  • 【Non-Slip Rubber Bottom】Featuring an anti-skid silicone base on the bottom, this wrist keyboard support stays firmly in place on your desk, preventing the padding from sliding around, ensuring stable and consistent wrist support during extended computer sessions.
  • 【Better Experience & Pain Relief】Our keyboard arm rest is beneficial to alleviate the soreness caused by direct contact and friction between your arm and a hard desk surface, reducing the risk of wrist fatigue or carpal tunnel. The soft texture of memory foam can evenly distribute the pressure around your wrists and provide good support with just enough give.
  • 【Helpful in Multiple Scenarios】Whether you're working, studying, writing, typing, gaming, this keyboard and mouse rest combo is an essential accessory to add comfort and support to your hands and wrists. It’s also a great gift for men, women, family, friend, coworker, gamer, teacher, etc.

Rate limiting is a burst-control tool, not a guarantee that an exact number of requests will reach the origin. Cloudflare warns that its rate-limiting rules are not designed to allow a precise number of requests to reach the origin. Counter updates can lag by seconds, so some excess requests may arrive before mitigation takes effect. Use caching and broader edge protection as the first line, then tune rate rules from observed traffic.

Free tools Windows power users keep installed

One-click scans. No signup required.

Special offer. See more information about Outbyte and uninstall instructions. Please review EULA and Privacy policy.

Safer tuning sequence

  1. Begin in a logging or preview mode when available and collect matches for a representative busy period.
  2. Set a threshold above the normal burst for legitimate users, monitors and verified crawlers.
  3. Challenge before blocking when the consequence of a false positive is high.
  4. Watch origin load, cache misses, response codes and integration errors after activation.
  5. Lower thresholds or extend mitigation only when repeated abusive behavior is confirmed.

robots.txt helps compliant crawlers, but it cannot protect the server

Add precise directives for crawlers that honor them, but do not mistake the file for an access-control system. Cloudflare’s June 2025 analysis calls robots.txt “an honor system.” A crawler can ignore it, spoof another user-agent or continue requesting URLs that the file disallows.

Cloudflare estimated that Google crawled about 14 times per referral, compared with roughly 1,700:1 for OpenAI and 73,000:1 for Anthropic. These are estimates; referrals from native applications may be unattributed. The ratios illustrate why a bot can generate substantial fetching without producing corresponding referral traffic.

A minimal preference file could look like this:

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

Publish only the directives that match your policy. Use edge rules, authentication and network controls for enforcement. If you want to permit a specific public section, replace the site-wide disallow with path-specific directives and verify the result in crawler documentation and logs.

Best Value
Sale
Under Desk Cable Management Tray No Drill, Metal Mesh Cable Management Under Desk with Clamp Mount, 2 Hole Cable Tray with Wire Management and Cord Organizer for Home Office Standing Desk
  • 【No Drill Mounted】Cable management tray can be assembled in as little as 3 minutes or less-simply. Side clips easily clampe the edge of desk, pass through the holes on the sides, and secure excess cables with baskets. Desk cable management supports inward or outward installation to meet your needs in different scenarios, and provides you with great convenience when collecting and organizing wires.
  • 【Rubber Pad Protects Desk】Wire organizer under desk built-in soft rubber pads on the clamp protect your desk from scratches, so you will not damage your furniture or office.
  • 【Sturdy and Beautiful】Cord management under desk is made of high-quality carbon steel and holds about 15 lbs. The wire basket has 2 holes on each side of the tray to enable your cables to pass freely. Cable baskets are suitable for desk thickness use from 0.4" to 2.4".
  • 【Space Saving & Keep Organized】Desk cable management tray holds all your power cords, outlet strips, USB hubs and computer transformers hidden from your feet, and the data cables will not enter the vacuum cleaner to keep your desk clean and tidy. Desk wire organizer is also suitable for use in the kitchen and outdoors.
  • 【Reduce Safety Risks】Cable tray under desk keeps all plugs away from your kids, no more worry about tripping. Using 2 holes wire tray saves extra space, helps you easily cabling and wire protection. The mesh design is not easy to accumulate dust, but avoids safety accidents caused by messy wires.
Independent reader supportYour contribution helps us test, update, and keep practical guides available for everyone.Support on Ko-Fi

Compare the available controls

Control Where it acts How selective it can be Origin cost and latency Main risk Observability and rollback
Full-page and object caching Edge and application cache URL, cookie, query and authorization rules Lowest cost for cache hits; cache misses still reach WordPress Stale or personalized content served incorrectly Use hit/miss metrics, purge controls and cache logs
Bot-score or managed challenge Edge Score, verified status, path, method and other request attributes Stops many requests before origin; challenges add client latency False positives for scripts, APIs or accessibility tools Review sampled matches and challenge outcomes; disable one rule to roll back
IP or ASN deny rule Edge or firewall Address ranges and networks Very low origin cost Shared addresses can include legitimate users; sources can rotate Maintain an allowlist and expiration review
Rate limiting Usually edge Expression, path, method, counter and time period Reduces bursts; counters can lag and are not exact Legitimate bursts throttled; verified bots may affect SEO Preview, tune thresholds and monitor 4xx/5xx rates
robots.txt Published site preference User-agent and path for compliant crawlers No direct origin protection Ignored or misread by noncompliant agents Inspect crawler behavior in logs; edit the file quickly
WordPress security plugin rule Origin application Often user-agent, IP, login and path rules Still consumes PHP and database resources before blocking Plugin overhead and conflicts during an attack Use plugin logs, staging tests and configuration backups

Protect SEO, customers and operational tools

Search optimization depends on access to the pages and resources that should be indexed. Allow verified Googlebot and Bingbot where they support your visibility, and avoid putting challenges on URLs that search systems cannot complete. Cloudflare cautions that rate-limiting verified bots can affect SEO.

Special offer. See more information about Outbyte and uninstall instructions. Please review EULA and Privacy policy.

After each rule change, compare search-console crawl activity and index coverage with your baseline. Check analytics referrals, real-user error reports, login success, checkout completion, API health checks and uptime-monitor results. A lower CPU graph is not a win if customers receive challenges at checkout or important pages disappear from search.

Keep a recovery path

  • Document every rule’s match expression, action, threshold, owner and creation date.
  • Keep an emergency allowlist for verified monitors, deployment systems and authenticated integrations.
  • Know how to disable a single edge rule without removing all protection.
  • Alert on rising 403, 429 and 5xx responses, origin saturation and search-crawl anomalies.
  • Recheck rules when WordPress, a major plugin, a CDN configuration or an API integration changes.

A practical rollout plan

  1. Measure for several normal traffic cycles. Build the bot, path, cache and resource baseline before making a decision.
  2. Fix cache coverage. Remove accidental cache bypasses and confirm that public content is served from the edge.
  3. Move enforcement outward. Put the CDN/WAF in front of the origin and close direct-origin bypasses where feasible.
  4. Allow verified and essential traffic. Define exceptions for search crawlers, monitors and authenticated integrations.
  5. Challenge suspicious requests. Start with low-risk public paths and bot-score or behavioral signals.
  6. Rate-limit expensive endpoints. Use narrow expressions for login, XML-RPC, search and APIs, then tune from logs.
  7. Block confirmed abuse. Deny persistent unwanted sources only after checking for shared addresses and false positives.
  8. Validate the outcome. Compare origin utilization, cache-hit rate, latency, errors, SEO signals and real-user journeys with the baseline.

The key decision is where a request is stopped. robots.txt communicates a preference, and an origin plugin may block late. Caching and edge controls prevent the largest amount of unnecessary WordPress work while preserving the ability to make selective exceptions for search, monitoring and legitimate users.

Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.