The Tool Desk
Outbyte PC Repair FREEClear out junk files and repair common Windows errorsFree Scan →Outbyte Driver Updater FREEFix the driver behind crashes, sound loss and screen glitchesFind Drivers →iTechGuides is reader-supported. When you buy through links on our site, we may earn an affiliate commission. As an Amazon Associate I earn from qualifying purchases. Learn more
Write one UTF-8 plain-text file at /robots.txt on each host you need to control, then create separate user-agent groups for search, training, and user-request crawlers. Test the deployed file—not just a local draft—against representative URLs and inspect your server logs after publishing.
A robots.txt policy is a voluntary crawler preference, not a security barrier. OpenAI, Google, Anthropic, and compliant third-party crawlers document different tokens and purposes, so blocking a training crawler does not necessarily remove your pages from search. The reliable approach is to identify each crawler’s purpose, state your policy explicitly, validate matching behavior, and use authentication or bot controls for anything that must be protected.
What does robots.txt control for AI crawlers?
robots.txt tells a crawler which URL paths it may access; it does not grant or revoke authorization. IETF RFC 9309 explicitly says that its rules are not access authorization, and Google describes the file as a way to tell search crawlers which URLs they can access. A crawler that chooses not to comply can still request the URL.
Quick wins for a faster PC:
Clear out junk files and repair common Windows errorsFree Scan →Fix the driver behind crashes, sound loss and screen glitchesFind Drivers →The file is evaluated per host, protocol, and port. A policy at https://www.example.com/robots.txt does not govern https://cdn.example.com/, another subdomain, a different port, or an HTTP endpoint. Use a separate root file for every host whose paths need a policy.
#1 Best Overall
- Powerful Turbo Fan:CAGIWIRU Electric Air Duster have reached speed up to 130000RPM,Can efficient remove dust and debris.With LED light and Three Adjustable Speed to meet with different cleaning tasks.
- With Attached USB C Fasting Charging cable.Charing 3 hours,Enjoy up to 240 minutes of use on the lowest setting, For third speed can use for 25mins.with four charging options to meet with your needs.
- Wide Application: attached 5 different nozzles,making it suitable for a variety of scenes, such as car,pet, pc, keyboards, and other electronic devices. It also serves for office and home clean duster.
- Handy Design & Portable:Cordless air duster electric nozzle is Light weight and small size, design and comfortable to hold and space saving, convenient to carry around. You can put it in your room or inside the car without taking up space.
- Economical &Reusable: Built durable materials with a long-lasting battery,our CAGIWIRU air duster it's a sustainable alternative to disposable air cans.
- Discovery and search: a search crawler may use the file to decide whether to fetch and index a URL.
- Model training: a vendor may provide a separate token whose disallow rule signals that content should not be collected for a stated training purpose.
- User-requested retrieval: an assistant may fetch a page because a person asked for it; that request can be governed separately, or a vendor may say robots.txt does not apply to that user-initiated action.
Because the mechanism is voluntary, treat robots.txt as a coordination and preference layer. For private records, paid material, unpublished work, or regulated data, enforce access with authentication, authorization, network controls, or a bot-management system as well.
Where must the file live, and what format does it require?
Publish one lowercase /robots.txt at the root of each relevant host. Serve it as UTF-8 plain text with the text/plain media type; do not create it in a word processor that can add smart quotes, hidden formatting, or a byte-order mark.
- Use a plain-text editor and ordinary LF or CRLF line endings.
- Keep the response at the exact root path, not in a subdirectory.
- Check every protocol, hostname, port, and subdomain that serves content.
- Keep the file below Google’s 500 KiB processing limit.
- Return a normal HTTP response without an HTML error page masquerading as text.
The policy applies only to URLs under the same host, protocol, and port. If both example.com and www.example.com serve pages, request both files directly and maintain both policies when their rules differ.
Which AI crawler tokens should I address?
Do not treat “AI crawler” as one category. Vendors expose independent user-agent tokens for search discovery, possible training collection, and user-triggered retrieval. The following purposes and effects are documented by the vendors as of October 2026.
| User-agent token | Documented purpose | What an Allow or Disallow preference means | Search visibility effect |
|---|---|---|---|
OAI-SearchBot |
Surfaces sites in ChatGPT search. | Allowing signals that the site may be fetched for search discovery; disallowing opts out of that crawler. | May reduce appearance in ChatGPT search, but it is independent of training controls. |
GPTBot |
Crawling that may contribute to training OpenAI foundation models. | Disallowing signals that content should not be used for that stated training purpose. | Does not by itself control ChatGPT search indexing. |
ChatGPT-User |
Retrieves a page in response to a user request. | OpenAI documents this separately and says robots.txt may not apply because the action is user initiated. | It is not used to determine ChatGPT search results. |
Google-Extended |
A product control token for use of Google-crawled content in future Gemini models, Gemini Apps, the Vertex AI API for Gemini, and grounding. | Allow or disallow it independently of Google’s ordinary search crawlers. | Google says it does not affect inclusion or ranking in Google Search. |
ClaudeBot |
Anthropic collection that may contribute to model training. | Anthropic says this bot honors robots.txt and documents an optional Crawl-delay extension. |
It is a training-collection control, not a guarantee about every Claude user retrieval. |
Claude-User |
Retrieves pages for a user’s Claude request. | It is documented separately from ClaudeBot; set a separate preference if your policy requires one. |
Blocking it can affect user-requested retrieval without being a training opt-out. |
Vendor names and purposes can change. Check each provider’s current crawler documentation before deploying a long-lived policy, and keep the groups separate so a decision about training does not accidentally remove search discovery.
How is a robots.txt rule written?
A group starts with one or more User-agent: lines and is followed by Allow: and Disallow: rules. Blank lines and comments beginning with # are safe for readability.
Rank #2
- Screen Cleaner is Your Screen’s New Bestfriend! - EVEO is proud to present a screen cleaner that’s perfectly suited for your TV. Meet your new screen cleaner, computer screen cleaner, laptop cleaner, iPad cleaner, Macbook,tv cleaner or any other electronic device. Our screen cleaner spray and wipe along with the included plush microfiber cloth, easily removes dust, fingerprints & other smudges on the screen’s surface. Clean effectively for a streak-free screen
- Gentle on your Screens - Our screen cleaner spray and wipes is compatible to be used even for the most sensitive LCD, LED, CRT, and OLED screens in proper use. Be at ease knowing that this product is compatible with all the major brands: LG, SONY, Samsung, Sharp, iPad, iPhone, Android screens, Kindle, PC monitors - we’ve got you covered! Can be used as LCD screen cleaner, tv cleaner, smart tv screen cleaner, and more
- Premium Super Soft Plush Microfiber Cloth - Included in this efficient screen cleaning kit. This screen cleaner spray and wipe has our signature plush microfiber cloth specially designed to comply with screens and monitors and leave them streak-free. Has been shown to effectively absorbs stubborn stains, zero streaks, and no fiber shedding. Always use this together with our screen cleaner for the best results
- Easy-to-Use Screen Cleaner Kit + microfiber cleaning cloth - The EVEO Screen Cleaning Kit will easily be your new favorite tool when it comes to taking care of the things you love most, quick and easy solution for a streak-free screen. Simply screw on the sprayer gun on the screen cleaner bottle. Spray the cleaning liquid onto the microfiber cloth, then wipe the screen until desired cleanliness & shine is achieved.
- Quality Screen Care Kit Worth Purchasing - Enjoy a convenient Screen Cleaner anytime, anywhere you need it. Your satisfaction is our top priority. We stand behind the quality of our products and that’s why, you shouldn’t be worrying at all. We manufacture the Best Screen Cleaner. and can be used for laptop cleaning kit, laptop screen cleaner, tv screen cleaner for smart tv. MacBook screen cleaner, monitor cleaner or screen cleaner spray for electronic devices with microfiber cleaning cloth
# One group per purpose
User-agent: OAI-SearchBot
Allow: /
User-agent: GPTBot
Disallow: /private/
Disallow: /drafts/
User-agent: Google-Extended
Disallow: /licensed-content/
User-agent: ClaudeBot
Disallow: /training-sensitive/
Sitemap: https://www.example.com/sitemap.xmlAn empty Disallow: value means no path is disallowed in that group. An Allow: / line makes the intention explicit, although a group with no disallow rules is already permissive.
How does matching decide between Allow and Disallow?
- User-agent token matching is case-insensitive.
- Path matching starts at the beginning of the URL path and is normally case-sensitive.
- The most specific matching path wins.
*matches zero or more characters.$anchors a match at the end of the path.- If Allow and Disallow rules are equivalent in length and specificity, Allow should win under RFC 9309.
For example, Disallow: /private/ covers that directory and descendants, while Disallow: /*.pdf$ targets paths ending in .pdf. Test wildcard and end-anchor behavior with the actual paths your site serves; a rule that looks right in prose can match more or fewer URLs than expected.
Should I combine duplicate groups?
Use one clearly consolidated group per token. Duplicate groups and conflicting copies make behavior harder to audit and can expose differences in parser implementations. Put every rule for a token in the same group, then test it as a unit.
Is Crawl-delay portable?
No. Anthropic documents Crawl-delay as a non-standard extension for its crawler. Do not assume Google, OpenAI, or another crawler will honor it; use ordinary rate limiting and server capacity controls when you need predictable traffic management.
How do I design a policy that preserves AI search visibility?
Separate discovery from training before writing any rule. A publisher that wants to appear in AI-powered search can allow the relevant search token while independently disallowing a training token.
- List the outcome you want. Decide whether each content area should be discoverable in search, available for user-requested answers, eligible for a vendor’s training collection, or none of these.
- Map the outcome to a vendor token. For OpenAI,
OAI-SearchBotandGPTBotare independent. For Google,Google-Extendedis a product-use control and not a separate Google Search crawler. - Write separate groups. Never use a broad wildcard group as a substitute for a purpose-specific decision unless you have intentionally chosen the same policy for every crawler.
- Scope sensitive paths. Use directory prefixes such as
/licensed-content/,/drafts/, or/training-sensitive/when the restriction is narrower than the whole site. - Document the rationale. A nearby comment can explain why a group exists, which helps future editors avoid deleting a deliberate distinction.
The skeleton above allows OpenAI search discovery, restricts selected paths for the potential training crawler, opts a licensed directory out of Google-Extended use, and restricts a training-sensitive directory for Anthropic. Adapt the paths and tokens to your own content inventory rather than copying the example unchanged.
Rank #3
- 【Large size】：Measuring 800 x 300 mm/ 31. 5×12 inches, which is wider and taller than others. Large mouse pad can perfectly fit your keyboard and mouse of any size, and there is also ample space for peripherals such as phones, tablets, etc.
- 【LED backlight mouse pad】：Blade Hawks RGB mouse pad provides 7 static modes and 7 dynamic modes will give you the options you are looking for. Power-off memory function remembers the last lighting mode you selected. The super glow fiber Chroma will give you a colorful gaming setup that you want.
- 【Anti-slip rubber base】：Made from Natural rubber, this gaming mousepad will firmly grip your desktop, allowing you to enjoy the ultimate precision in the games you are most passionate about. It also comes with a texture that enhances this ability even further.
- 【Smooth and waterproof surface】: Micro-textured cloth surface, for extremely precise mouse control and a smooth movement. When liquid splashes on the gaming mouse pad, it forms water droplets and slides down. Will not delay your work or gameplay.
- 【Plug and play】: This mouse and keyboard pad is powered by USB, plug and play, no driver required. One button control light modes. Press one time to change the lighting mode, press twice to change the brightness, press and hold about 3 seconds turn ON/OFF.
How do I block a specific AI crawler or directory?
Place the exact token in its own group and add only the paths you intend to block. To block that token everywhere, use Disallow: /; to block a directory and its descendants, end the path with a slash.
User-agent: GPTBot
Disallow: /
User-agent: OAI-SearchBot
Allow: /
User-agent: ClaudeBot
Disallow: /unpublished/
Disallow: /client-material/A broad block can remove a site from the corresponding crawler’s discovery even when ordinary search engines remain unaffected. Review the visibility consequence for each token before publishing. Also remember that a non-compliant scraper can ignore the rule entirely; a disallow line cannot prevent a direct HTTP request.
How do I block file types or exact URLs?
Use wildcard and end-anchor patterns carefully. For example, Disallow: /*.pdf$ matches paths ending in .pdf, while Disallow: /reports/ covers the entire reports directory. If an allowed exception is at least as specific as a disallow rule, test the tie explicitly because an equal-strength Allow should win.
What’s actually slowing this PC down?
Pick the symptom - the matching free tool is one click away.
How do I test a deployed robots.txt file?
Testing must cover the live response, parser behavior, and real URL patterns. A file that passes a local text check can still be unavailable at the production hostname, cached, redirected, encoded incorrectly, or overridden by a firewall.
- Validate syntax and encoding. Confirm UTF-8 plain text, valid
field:valuelines, ordinary line endings, no smart quotes, no byte-order mark, and a size below 500 KiB. - Fetch every live endpoint. Open
https://host/robots.txtin a private browser window and inspect the HTTP status, content type, redirects, and body. Repeat for each protocol, hostname, port, and subdomain that needs a policy. - Check redirect behavior. RFC 9309 says crawlers should follow at least five redirects. Test the final file and remove redirect chains that are unnecessary or inconsistent between hosts.
- Run representative matches. For every AI token, test an allowed URL, a disallowed URL, a nested path, a wildcard match, an end-anchored match, and an Allow/Disallow tie.
- Use Google’s tools where relevant. Search Console provides a robots.txt report for supported properties, and Google publishes an open-source robots.txt library that can evaluate the file against representative URLs.
- Inspect logs after release. Look for the documented user-agent tokens, the requested paths, response codes, and whether a CDN, firewall, or bot manager blocked the fetch.
What should the test matrix contain?
| Test case | Example path or condition | Expected verification |
|---|---|---|
| Explicit allow | /public/article |
The relevant token has an Allow rule or no matching Disallow. |
| Directory block | /private/record.html |
The directory rule wins for the file and its descendants. |
| Nested exception | An allowed child below a blocked parent | The more specific Allow wins when its match is equal or more specific. |
| Wildcard | A URL containing the wildcard pattern | The parser matches the intended characters and no unintended paths. |
| End anchor | A path ending and not ending with the anchored suffix | $ limits the rule to the intended endpoint. |
| Duplicate definition | Repeated token groups | The deployed file has been consolidated and produces deterministic results. |
A direct fetch proves what the public internet receives; a local parser test proves how a standards-aware implementation interprets it. Use both, then compare the result with logs from real requests.
What happens when robots.txt is redirected, missing, or malformed?
Redirects
Keep redirects short and unambiguous. Crawlers should follow at least five redirects under RFC 9309, but a long chain increases the chance of failure and stale interpretations. Make sure the final response is the intended plain-text file for the same host policy.
Rank #4
- 【Softer and More Comfortable】Vaydeer wrist rest has unique diamond pattern, which is the combination of softness and aesthetics. The materials of wrist rest are improved into higher quality memory foam and covered with silky smooth lycra. The computer wrist rest makes you as comfortable and cushiony as like rest your wrists on clouds.
- 【Ergonomic Wrist Saver】The wrist rests for keyboard and mouse comes with a 17.32×3.15×0.83 inch keyboard wrist pad and a 5.94×3.15×0.83 inch mouse wrist support. Based on ergonomic design, the unique concave shape is the perfect fit for your wrist joints. The wrist rest pad fits most computer keyboards and laptops, improve hand and wrist posture, release your wrist and arm stress.
- 【Non-Slip Rubber Bottom】Featuring an anti-skid silicone base on the bottom, this wrist keyboard support stays firmly in place on your desk, preventing the padding from sliding around, ensuring stable and consistent wrist support during extended computer sessions.
- 【Better Experience & Pain Relief】Our keyboard arm rest is beneficial to alleviate the soreness caused by direct contact and friction between your arm and a hard desk surface, reducing the risk of wrist fatigue or carpal tunnel. The soft texture of memory foam can evenly distribute the pressure around your wrists and provide good support with just enough give.
- 【Helpful in Multiple Scenarios】Whether you're working, studying, writing, typing, gaming, this keyboard and mouse rest combo is an essential accessory to add comfort and support to your hands and wrists. It’s also a great gift for men, women, family, friend, coworker, gamer, teacher, etc.
4xx responses
An unavailable-file response in the 4xx range can be treated as if no robots policy exists, depending on the crawler. That may allow crawling, so do not rely on a missing file as a block. Monitor status codes and restore a valid file quickly after an accidental 404 or 410.
5xx responses and network failures
RFC 9309 calls for complete disallow when the file is unreachable because of server or network failure, but implementations can differ while recovering. Google may retain a cached copy for up to 24 hours or longer when it cannot refresh the file. Design deployments so a temporary outage does not create an unexpected policy change, and verify behavior after service restoration.
Malformed lines and encoding errors
Invalid field separators, smart punctuation, a byte-order mark, or an HTML error document can cause rules to be ignored or parsed differently. Keep syntax simple, inspect the raw response, and run the open-source parser against the exact deployed bytes.
Oversized files and stale caches
Google’s documented processing limit is 500 KiB. Large generated files can be truncated at that boundary, dropping rules near the end. After a change, check cache headers and fetch from an external network; a browser showing a new file does not prove every crawler has refreshed its copy.
Independent reader supportYour contribution helps us test, update, and keep practical guides available for everyone.Can robots.txt stop a non-compliant scraper?
No. A crawler that ignores the Robots Exclusion Protocol can fetch disallowed URLs, and robots.txt contains no authentication or permission check. Protect confidential or commercially restricted content with login requirements, authorization checks, signed URLs, network policy, or a bot-management control, then use robots.txt as an additional signal for compliant crawlers.
Logs help distinguish a policy problem from abuse. Record the source address, claimed user-agent, requested path, response code, and rate. User-agent strings can be forged, so do not grant sensitive access solely because a request claims to be an approved crawler.
Best Value
- 【No Drill Mounted】Cable management tray can be assembled in as little as 3 minutes or less-simply. Side clips easily clampe the edge of desk, pass through the holes on the sides, and secure excess cables with baskets. Desk cable management supports inward or outward installation to meet your needs in different scenarios, and provides you with great convenience when collecting and organizing wires.
- 【Rubber Pad Protects Desk】Wire organizer under desk built-in soft rubber pads on the clamp protect your desk from scratches, so you will not damage your furniture or office.
- 【Sturdy and Beautiful】Cord management under desk is made of high-quality carbon steel and holds about 15 lbs. The wire basket has 2 holes on each side of the tray to enable your cables to pass freely. Cable baskets are suitable for desk thickness use from 0.4" to 2.4".
- 【Space Saving & Keep Organized】Desk cable management tray holds all your power cords, outlet strips, USB hubs and computer transformers hidden from your feet, and the data cables will not enter the vacuum cleaner to keep your desk clean and tidy. Desk wire organizer is also suitable for use in the kitchen and outdoors.
- 【Reduce Safety Risks】Cable tray under desk keeps all plugs away from your kids, no more worry about tripping. Using 2 holes wire tray saves extra space, helps you easily cabling and wire protection. The mesh design is not easy to accumulate dust, but avoids safety accidents caused by messy wires.
How should I maintain AI crawler rules over time?
- Keep an inventory of every host and subdomain that serves crawlable content.
- Review vendor documentation before adding or removing a token; names and purposes can change.
- Store the policy in version control and require review for broad changes such as
Disallow: /. - Automate checks for UTF-8, file size, duplicate groups, malformed lines, and accidental HTML responses.
- Run representative matching tests whenever URL structures, redirects, or content classifications change.
- Check Search Console reports where available and inspect logs after deployment.
- Alert on unexpected 4xx or 5xx responses, redirect loops, cache-age anomalies, and sudden crawler traffic changes.
- Keep comments focused on policy intent, not temporary implementation details that will become misleading.
Re-test after CDN, firewall, hosting, or domain migrations. A technically correct file on the old host does not govern the new hostname until that host serves its own root file.
Which policy approach fits my publishing goal?
| Goal | Typical policy choice | Main trade-off |
|---|---|---|
| Maximize AI search discovery | Allow the relevant search token, such as OAI-SearchBot, while reviewing other tokens independently. |
Search visibility is preserved, but separate training crawlers may still need their own decision. |
| Opt out of a documented training use | Disallow the vendor’s training token, such as GPTBot, Google-Extended, or ClaudeBot, according to the vendor’s stated scope. |
The preference is voluntary and does not retroactively control content already collected. |
| Restrict only licensed or unpublished paths | Use path-specific Disallow rules instead of blocking the whole host. | URL changes can create gaps unless tests and redirects are maintained. |
| Protect confidential material | Use authentication or authorization first; add robots.txt as a compliant-crawler signal. | Requires access-control engineering rather than relying on a text file. |
| Reduce request pressure | Use server-side rate limits and capacity controls; consider documented extensions only where supported. | Non-standard directives such as Crawl-delay are not portable. |
What is the practical deployment checklist?
- Identify every host, protocol, port, and subdomain that needs a policy.
- Classify each AI token as search, training, or user-request retrieval.
- Write one UTF-8 plain-text file at lowercase
/robots.txtfor each host. - Use separate, consolidated groups and path rules that reflect your content inventory.
- Check wildcard, end-anchor, nested, and equal-specificity matches.
- Confirm the response is below 500 KiB, served as
text/plain, and free of encoding artifacts. - Fetch the live URL, inspect redirects and status codes, and run a standards-aware parser.
- Use Search Console’s report where available and review production logs after release.
- Keep private content behind access controls because robots.txt is not authorization.
Frequently Asked Questions
Does changing robots.txt remove content an AI model already collected?
No. A new rule communicates a preference for future crawling; it does not erase copies or model training that may already have occurred. If a provider offers a separate removal or data-rights process, that process is distinct from robots.txt.
Should a staging site have a robots.txt block?
A staging environment should normally require authentication or network restriction, because a public disallow file does not hide it from people or non-compliant bots. You can add a restrictive robots.txt as a precaution after access control is in place.
Outdated Drivers Are Slowing You Down
One free scan finds every outdated or missing driver and matches the right update for your exact hardware.Free scan · exact hardware matchPC Slower Than It Used to Be?
A free scan shows the junk files, broken settings and background clutter dragging Windows down - then fixes them in one click.Free scan · Windows 10 & 11What should I do if robots.txt returns 404 intermittently?
Treat the outage as an operational incident, not as an intentional policy. Check routing, CDN rules, deploy artifacts, and origin health, then fetch the file repeatedly from outside your network until the status and body are stable.
Do robots.txt rules apply to image, video, and API URLs?
They apply to URL paths regardless of whether the response is HTML, media, or JSON. Test the actual endpoint paths, including versioned API routes and query-handling behavior, because a rule written for page directories may not cover those URLs.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
