In SerpPrism’s hand-picked sample of 70 usable robots.txt files, explicit rules for AI crawlers were uncommon; when sites named crawlers, most blocked at least one. These are the rules the files stated—not evidence that crawlers followed them or that the same pattern holds across the wider web.
What the 70-file sample found
SerpPrism maintainer Hongtao Ren fetched robots.txt files for 78 well-known domains on September 23, 2026. The study parsed 70 usable files and checked 22 selected AI crawler tokens to see whether each could fetch the site’s root path under the file’s rules. For a token without its own group, the analysis applied the wildcard group. It classified results as blocked, partially restricted on some paths while the root remained open, or allowed. The author says path matching used Google’s rule that the longest matching pattern wins, with an equal-length Allow taking precedence. SerpPrism’s study.
The 78 domains spanned news, commerce, SaaS, developer tools, education, and government, but were selected as well-known sites rather than drawn randomly. Eight files could not be used: Stack Overflow returned HTTP 418; npmjs.com and nih.gov returned 403; Vimeo timed out; rust-lang.org and wikimedia.org returned 404; and Khan Academy and CDC returned HTML error pages with HTTP 200.
| Finding in SerpPrism’s selected sample | Count |
|---|---|
| Usable robots.txt files analyzed | 70, fetched September 23, 2026 |
| Files with no named AI crawler token | 44 of 70 |
| Files naming at least one AI crawler | 26 of 70 |
| Named-crawler files blocking at least one named crawler | 22 of 26 |
| Named-crawler files blocking every AI crawler they named | 13 of 26 |
Among the 44 files without an AI-specific token, 42 allowed crawlers through the wildcard policy. Reddit and Pinterest instead used wildcard Disallow: /, which blocks all crawlers covered by that group. An absent AI-specific rule therefore does not by itself establish whether a crawler is allowed.
#1 Best Overall
- 【Powerful Load-bearing】12U Network Rack Open Frame is constructed from durable cold rolled steel; Rack shelf supports enhance stability, wall-mounted capacity of 130lbs, the ground-mounted up to 260lbs
- 【Considerate Designs】Open-frame layout, including a top panel adding space, anti-slip shelf stops fixing devices and compatible racks for stack and expansion to meet requirements of home server rack
- 【Complete Accessories】A 12U open frame server rack, two ventilated shelves, four shelf stops, four velcro straps and a set of equipment mounting screws
- 【Versatile Application】Ideal for space-efficient multi-device setups in warehouses, retail, classrooms, offices and more; Excellent choices as AV Rack/IT Rack
- 【Effortless Setup】 Network Rack includes hardware, a comprehensive manual, mounting hole drilling template and an online assembly video to simplify setup
Which named crawlers were blocked most often?
The figures below count only sites in this sample that named each token. They are not rates for the web as a whole.
| Crawler token | Named by | Blocked by | Share blocked among naming sites |
|---|---|---|---|
| GPTBot | 17 sites | 10 | 59% |
| ClaudeBot | 18 sites | 13 | 72% |
| CCBot | 15 sites | 13 | 87% |
| Bytespider | 11 sites | 11 | 100% |
| Applebot-Extended | 10 sites | 10 | 100% |
| omgili | 9 sites | 9 | 100% |
| Diffbot | 8 sites | 8 | 100% |
The 100% figures mean every site naming that token in this 70-file sample blocked it; the denominators are small, and they do not establish how sites generally treat those crawlers.
Rank #2
- Adjustable Depth: 23-40'' adjustable depth is used for servers and network equipment, ensuring enough space for AV equipment, components, and cabling, while allowing you to access ports and equipment from multiple sides.
- Strong Load Capacity: Ground-Mounted Load Capacity: 500 lbs, Wall-Mounted Load Capacity: 150 lbs. The av rack is made of carbon steel for better weldability performance and can help save space while meeting your need to place multiple devices.
- User-friendly Design: Ergonomic design makes the open frame av rack easier to use. The additional top panel is able to place other items with more available space. Roller design moves anywhere and anytime, is convenient, and is more energy-saving.
- Complete Accessories: We provide the accessories you need, including 2 x Pallets, 145 x M5*10 Cross Head Screws, 4 x Casters, 4 x M10*50 Expansion Screws,10 x M6*12 Cage Nuts, 1 x Grounding Wire, 1 x User Manual.
- Wide Application: The server rack wall mount maximizes the use of available space, suitable for retail venues, classrooms, offices, and other places where space is limited.
Why a robots.txt policy needs more than an allow-or-block label
The examples show that a policy can depend on the crawler, the path, and whether multiple tokens share a directive group. SerpPrism found four files that named crawlers but blocked none of the named tokens: Cloudflare, Netlify, Moz, and Twitch. Twitch’s empty Amazonbot directive is an allow rule in effect: Disallow: with no path disallows nothing.
Different purposes can have different rules
Cloudflare’s file acts as an affirmative allowlist for several AI crawler tokens and includes a comment about permitting access to markdown versions of pages. Netflix takes a different approach: a wildcard disallow paired with an allowlist that includes named search and AI crawlers. These examples illustrate why a site-wide rule for every crawler may not match a publisher’s intended policy.
What’s actually slowing this PC down?
Pick the symptom - the matching free tool is one click away.
Rank #3
- Adjustable Depth: Depth adjustable from 23" to 40", this open frame server rack accommodates servers and network equipment while providing ample space for A/V gears and cable management. Enjoy easy access to ports and devices from multiple angles.
- High Weight Capacity: Supports up to 300 lbs on the floor (200 lbs when adjusted to maximum depth) and 200 lbs when wall-mounted (depth cannot be adjusted in wall-mounted mode). Made from carbon steel for superior welding performance and durability, this open frame rack is designed to save space while accommodating multiple devices.
- User-Friendly Design: Designed with your convenience in mind, this open frame server rack features an top shelf for extra storage and improved space utilization. The rolling casters let you move it effortlessly wherever you need it, making setup and movement a breeze.
- Widely Applicable: Maximize your space with this adaptable open frame server rack, designed to make the most of every inch. Ideal for retail spots, classrooms, offices, and any area where space is at a premium, it delivers practical solutions for your storage needs.
- Everything You Need: Our open-frame rack comes with fully equipped accessory kit for easy setup and secure installation: 2 x Trays, 4 x Casters, 1 x set of Screws, 16 x M6*12 Cage Nuts, 1 x Grounding Wire, 1 x Internal & External Hex Wrenches, and 1 x User Manual.
Some files distinguish training crawlers from search or user-triggered retrieval tokens. eBay’s rules differ for GPTBot and Applebot-Extended compared with OAI-SearchBot, ChatGPT-User, Claude-SearchBot, and Claude-User. LinkedIn also treats GPTBot, ChatGPT-User, and OAI-SearchBot differently. SerpPrism’s examples show distinctions in the file rules; they do not independently establish current product behavior or guarantee what a service will display.
Path-specific and shared rules can change the result
Moz blocks GPTBot from /blog/ and /learn/seo/, rather than applying that restriction to every path. GitHub places several crawler tokens in a shared group with a crawl delay and a marketing-page allowlist, while giving Bytespider a separate block. A change to a shared group can affect every crawler named in it, so check the group membership before editing a directive.
Quick Recap
Best Value
- Standard 1U Height: Get more space with our 1U server rack shelf—it comes in a set of 4! Ideal for 19-inch 4-post server racks, stacking routers, switches, firewalls, and other network gear. Easy storage and a neat setup in one simple solution
- Heavy-Duty Construction: Crafted from premium Q235 carbon steel with a robust 0.06 in (1.5 mm) thickness, our network rack shelf can handle up to 50 lbs (22.68 kg) with ease. Say goodbye to wobbles and tilts—keeping everything in its place
- Optimal Ventilation: Featuring a vented bottom design, our rack mount shelf effectively reduces equipment temperature, ensuring stable operation and lowering the risk of malfunctions. Keep your gear running smoothly for longer-lasting performance
- Flexible Partitioning: Each shelf features a depth of 10 in (254 mm). Our server rack shelf helps you organize and optimize your rack space efficiently. Keep your equipment neatly separated to reduce clutter and minimize interference or collisions
- Installation Made Easy: Everything you need for installation is included—screws and nuts are provided, making the process quick and hassle-free. Simply use a Phillips screwdriver, and you'll have your network rack shelf installed in no time
Rank #4
- ADJUSTABLE DEPTH: 4-Post 42U open frame server rack with 4 vertical rails and adjustable mounting depth 22" to 40" (56,0cm to 101,7cm); Compatible with various servers / switches / data / AV and other IT equipment; EIA/ECA-310-E Compliant
- EASY ASSEMBLY: Mobile network rack with easy-to-follow assembly instructions and online video; Compact flat-pack shipping to avoid damage and facilitate installation; Total product height of 80.3in (204 cm) with casters, 78in (198cm) without casters
- COLD ROLLED STEEL: Durable 4 Post 19in open frame rack designed for ventilation with 42U mounting height and 1320lb (600kg) weight capacity (stationary); 3 install options included: casters, levelling feet, or base-plate to secure rack to the floor
- HARDWARE INCLUDED: Rolling computer/data rack includes cage nuts and screws to mount equipment, easy to read Units (U) and depth adjustment markings, cable management hooks for organization, and required assembly tools
- THE IT PRO'S CHOICE: Designed and built for IT Professionals, this 42U rack is backed for 2-years, including free lifetime 24/5 multi-lingual technical assistance
How to check what a crawler is allowed to fetch
- Choose the crawler and purpose. Identify the exact token and decide whether your policy should distinguish training-oriented crawling from search or user-triggered retrieval. Do not assume all AI-related tokens serve the same purpose.
- Read the complete robots.txt file. Find the crawler’s specific group, if any, and compare it with the wildcard group. An unnamed crawler may inherit wildcard directives; silence in a crawler-specific group is not enough to determine the effective rule.
- Resolve matching rules for real URLs. Check the homepage, an article, and a path you believe is restricted. Apply the most specific matching pattern; in SerpPrism’s analysis, the longest matching path wins, with an equal-length
Allowtaking precedence. - Use a robots.txt tester. Test the exact crawler token against multiple representative URLs. A result for the root path does not tell you whether every other path is accessible under the file.
- Separate stated policy from access control. Robots.txt is advisory. Firewalls, rate limits, terms of service, and a crawler’s decision not to comply fall outside what the file can establish. As Ren puts it, “robots.txt is a request, not a lock.”
What this study does—and does not—establish
- A selected sample, not a web-wide estimate: the domains were hand-picked and the counts describe only 70 usable files.
- A limited set of tokens: the analysis covered 22 selected documented crawler tokens; it cannot speak for unexamined or later-added tokens.
- A one-day snapshot: the files were fetched September 23, 2026, and may change.
- File policy, not observed behavior: robots.txt alone cannot show whether a crawler obeyed a rule or whether a site restricts access through another layer.
- Sample-specific non-findings: the study found no misspelled AI crawler tokens in its sample. It found no site blocking Googlebot except Reddit, whose wildcard rule blocks everyone. These findings do not show that such cases are absent elsewhere.
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
