The Tool Desk
Outbyte PC Repair FREEClear out junk files and repair common Windows errorsFree Scan →Outbyte Driver Updater FREEFix the driver behind crashes, sound loss and screen glitchesFind Drivers →iTechGuides is reader-supported. When you buy through links on our site, we may earn an affiliate commission. As an Amazon Associate I earn from qualifying purchases. Learn more
robots.txt tells compliant crawlers which URL paths they may request; it does not make those paths private or guarantee they will stay out of Google Search. To remove a crawlable page from Google, let Google access it and serve a noindex directive. To keep content private, require authentication or another access-control measure.
What robots.txt does
robots.txt is a public, plain-text file that communicates crawler access preferences for paths on a website. The Robots Exclusion Protocol is standardized in RFC 9309, published by the Internet Engineering Task Force in September 2022. The standard is explicit: “These rules are not a form of access authorization.”
A crawler that honors the file uses its rules to decide which paths to request. This makes robots.txt useful for managing crawling, but it does not control what people can access or guarantee how a search engine indexes a URL.
What robots.txt cannot do
It cannot guarantee a URL stays out of Google
Google says robots.txt is not a way to keep a web page out of Google Search. If Google discovers a blocked URL through links, it may index the URL without crawling its contents; the result may appear without a snippet. See Google’s robots.txt guidance.
It cannot make content private
The file is publicly accessible, and listing a path can draw attention to it. Anyone can still try to visit a disallowed URL directly. If access must be restricted, use authentication or another application- or server-layer control, as RFC 9309 recommends.
It cannot deliver a noindex directive to a blocked page
Google must be able to crawl a page or resource to read its robots meta tag or X-Robots-Tag header. If robots.txt blocks the request, Google cannot see the directive. Google explains this in its robots meta tag and X-Robots-Tag documentation.
Choose the control that matches your goal
| Goal | Use | Important limitation |
|---|---|---|
| Manage crawler requests to URL paths | robots.txt Allow and Disallow rules |
Does not secure content or guarantee search removal. |
| Keep a crawlable page out of Google Search | A robots meta noindex directive or an X-Robots-Tag: noindex header |
Google must be able to crawl the page to read the instruction. |
| Restrict access to sensitive content | Authentication or another access-control mechanism | Must be implemented at the application or server layer, not in a public robots.txt file. |
These mechanisms address different problems. A disallow rule can reduce crawling, but it is not a substitute for either a search-indexing directive or access control.
Where to put the file and how it is interpreted
RFC 9309 specifies a file named /robots.txt at the top level of a service. It should be UTF-8 encoded and served as text/plain. The rules apply to the service’s paths; placing a file in a subdirectory does not make it the service’s root robots.txt.
Rules are grouped by crawler user agent and use directives such as Allow and Disallow. Under RFC guidance, path matching is case-sensitive and starts at the first character of the path. The most specific matching rule takes precedence; when an allow and disallow rule are equally specific, the standard says the allow rule should be used. Individual crawlers may differ in how they implement extensions, so do not assume every crawler interprets every feature identically.
Unavailable and unreachable files are different cases
RFC 9309 distinguishes a robots.txt response that is unavailable, such as an HTTP 4xx, from one that is unreachable because of server or network errors. For an unavailable response, a crawler may access resources. For an unreachable file, the RFC says crawlers must assume complete disallow, subject to provisions for prolonged failures and cached copies. These are standard recommendations for crawler behavior, not a guarantee that every implementation will act identically.
Rank #4
The RFC also says crawlers should not use cached robots.txt content for more than 24 hours unless the file is unreachable.
Quick wins for a faster PC:
Scan for outdated or missing drivers - takes under a minuteDriver Scan →Repair Windows errors before they cause bigger problemsFix Now →Adding a sitemap reference
A robots.txt file can point crawlers to a sitemap with a fully qualified URL, for example:
Sitemap: https://example.com/sitemap.xmlGoogle’s robots.txt specification guidance says the sitemap URL can be on another host and that multiple Sitemap: fields are supported. A sitemap reference helps discovery; it does not override access rules or guarantee indexing.
Quick Recap
Practical checklist
- Use robots.txt when the goal is to manage crawler requests to paths, not to hide information.
- For Google search removal, allow crawling and use a robots meta
noindexdirective or anX-Robots-Tag: noindexheader. - For private content, require authentication or apply another access-control measure.
- Put the file at the service root, and verify that its content is served as UTF-8 plain text.
- If you include a sitemap, use its full URL in a
Sitemap:field.
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
