iTechGuides is reader-supported. When you buy through links on our site, we may earn an affiliate commission. As an Amazon Associate I earn from qualifying purchases. Learn more
The change is a shift from treating bots as one category to classifying automated access by purpose. That distinction matters because a single crawler can serve more than one purpose, so blocking it outright may also affect search access.
What changed in how bots are classified?
Cloudflare’s September 2026 controls group automated activity into three behaviors: Search, Training, and Agent. This is Cloudflare’s way of organizing its own controls, not a universal web standard. A crawler’s label describes a purpose; it does not necessarily mean that the crawler has only that purpose.
| Behavior in Cloudflare’s model | What the label distinguishes | Why it matters to a site owner |
|---|---|---|
| Search | Automated access associated with search | A block may affect whether a search crawler can access pages. |
| Training | Automated access associated with AI training | A site owner may want to express a training preference without cutting off search access. |
| Agent | Automated access classified as agent behavior | It is a separate purpose category in Cloudflare’s model; the label alone does not establish that a crawler has no other purpose. |
Can you block AI training bots without blocking search?
Sometimes—but the outcome depends on the crawler and the control used. Cloudflare distinguishes a published preference from a block. Its Disallow AI Training preference is the route it describes for keeping accountable mixed-use crawlers available for search. A full block is a separate control and can cut off search access when applied to a crawler that also serves search.
Quick wins for a faster PC:
Repair Windows errors before they cause bigger problemsFix Now →Fix the driver behind crashes, sound loss and screen glitchesFind Drivers →Clear out junk files and repair common Windows errorsFree Scan →Cloudflare’s September 2026 examples include Applebot, Bingbot, and Googlebot as mixed-use crawlers that a full block can affect. So the practical question is not only “Is this an AI bot?” but also “What other access does this crawler provide, and will this control prevent it?”
Robots.txt expresses a site’s preference; it is not a technical enforcement mechanism. Cloudflare’s blocking controls operate separately. A preference and an enforced block should not be treated as interchangeable.
What if one crawler does more than one thing?
A mixed-use crawler creates a trade-off: blocking it to limit one purpose can also restrict another. For example, a full block on a crawler used for both training and search can affect search access as well as training. If preserving search access is important, Cloudflare describes its training-specific preference as the option for keeping accountable mixed-use crawlers available for search.
There can also be limits to how finely a site can distinguish crawler variants. Google documents that Googlebot Smartphone and Googlebot Desktop share the same robots.txt product token. A rule keyed to that token therefore does not, by itself, distinguish those two subtypes.
PC Slower Than It Used to Be?
A free scan shows the junk files, broken settings and background clutter dragging Windows down - then fixes them in one click.Free scan · Windows 10 & 11Crashes, No Sound, or Screen Glitches?
Random freezes, missing sound and display glitches usually trace back to one bad driver. Find and replace yours safely.Free scan · under a minuteHow large are these crawler categories?
Cloudflare’s worldwide verified-bot data for 2025 gives a sense of activity on its own network—not a census of all web traffic or of every crawler on the internet:
Rank #3
- 【Tired of constantly searching for or resetting your passwords?】 MOSA BEAR password keeper book is the perfect solution for you! This password book provides a dedicated place to securely store all your important website addresses, emails, usernames and passwords, ensuring your information is protected and easy to find. The well-designed log pages help you manage multiple accounts in a systematic way, saying goodbye to password confusion.
- 【Premium Design & Password Security】 The password book with alphabetical tabs features an anonymous cover design with no title on the cover, effectively avoiding information exposure. The password keeper design is specifically designed with password security in mind, providing space to record password hints instead of writing directly on the password itself, further protecting your important information.
- 【Simple Layout and Plenty of Space】The 160-page password logbook is designed to provide ample space to record passwords and other important information. It can store up to 414 passwords. In addition, it provides extra pages to record other information, such as email setup, card information, computer operating system information, software licenses, and more. The journal also includes 3 blank pages at the end for you to add additional notes.
- 【Palm-sized Size & Premium Quality】 This password notebook has an ideal size, 4.3" x 5.7", for carrying around, whether in a purse or pocket. Its sturdy glue binding allows the notebook to unfold smoothly and is more comfortable to use. The inner pages are made of high-quality 100GSM thick paper, which can effectively reduce ink penetration and ensure a cleaner and neater writing effect. The overall design takes into account both portability and durability, making it an ideal choice for recording important passwords.
- 【A-Z Tabs for Quick Search 】Our password book comes with alphabetical tabs to help you find the password you need quickly and easily. Alphabetically organized tabs ensure that you can quickly flip to the right section, saving you the time and hassle of searching for your password.
- Search crawlers accounted for 40% of verified-bot traffic; AI crawlers accounted for 20%.
- Googlebot made up more than 28% of verified-bot traffic, GPTBot about 7.5%, and Bingbot 6%.
- Cloudflare reported that ChatGPT-User request volume peaked at up to 16 times its beginning-of-year level in 2025. OAI-SearchBot’s late-October 2025 peak was about five times its beginning-of-year level.
Those last figures describe measured crawler requests, not numbers of people using a service. They also show why a bot’s function and its request volume are different questions: a rise in requests does not reveal how many human users prompted them.
Independent reader supportYour contribution helps us test, update, and keep practical guides available for everyone.What should a site owner choose?
Start with the outcome you want, then check the crawler’s other purposes before choosing a control. In Cloudflare’s system, a training preference and a full block have different consequences:
Rank #4
- Bookbound planner helps you keep track of passwords and favorite websites
- Room for over 200 entries; 3.5 x 6 inch page sizes
- User name and security questions field
- Tips for what makes a strong password; web resources; notes pages
- Printed on quality paper containing 30% post-consumer waste; black simulated leather cover; 3.63 x 6.13 x .21 inches
- If you want to express a training preference: Cloudflare describes Disallow AI Training as the option for keeping accountable mixed-use crawlers available for search. This is a preference, not the same thing as an enforced block.
- If you want to prevent a crawler from accessing the site: use a blocking control, while accounting for any search access that crawler also provides.
- If a crawler has multiple purposes or subtypes: verify how Cloudflare classifies it and what the selected control applies to; do not assume that a purpose label or robots.txt token isolates every use or variant.
Cloudflare’s September 2026 recommendations for new domains differ depending on whether a site is ad-supported, and customers can change the recommended setting. Treat those defaults as Cloudflare’s guidance for its product, not a rule all sites must follow.
The Tool Desk
Outbyte Driver Updater FREEFix the driver behind crashes, sound loss and screen glitchesFind Drivers →Outbyte PC Repair FREEClear out junk files and repair common Windows errorsFree Scan →Cloudflare co-founder and CEO Matthew Prince described the goal this way: “This is how we make the Internet better: preserving the openness that makes search valuable while giving the people and businesses behind the web meaningful control over how their work is used.”
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
