robots.txt communicates crawler access preferences, sitemap.xml helps search engines discover URLs, and llms.txt is a proposal for giving language-model tools a curated overview of a site. They solve different problems and none replaces the others. In particular, robots rules do not protect private files, a sitemap does not guarantee indexing, and Google says llms.txt does not affect Google Search visibility or rankings.
How the three files differ
| File | Main purpose | Intended consumer | Typical contents | What it cannot promise |
|---|---|---|---|---|
robots.txt |
Communicates crawler access preferences for paths on a site. | Automated crawlers. | User-agent groups and Allow/Disallow rules; it may also include a Sitemap record. |
It does not authorize or prevent human access, and is not a security mechanism. [IETF RFC 9309] |
sitemap.xml |
Helps search engines discover URLs and associated information. | Search engines. | URLs, with optional metadata or extensions such as last-modified dates, images, video, news, and alternate language versions. | Listing a URL does not guarantee that it will be crawled, indexed, or shown in results. [Google Search Central] |
llms.txt |
Proposes a concise, curated overview of a website for LLM agents to consult on demand. | LLM agents or tools that choose to use it. | A site overview and selected links or material. | It is a proposal, not a crawler permission file or complete search URL list. Google says it has no effect on Google Search visibility or rankings. [llms.txt proposal] [Google Search documentation update, June 15, 2026] |
The practical distinction is access preferences (robots.txt), URL discovery (sitemap.xml), and curated context for compatible LLM tools (llms.txt). You can maintain more than one because they serve different roles.
| # | Preview | Product | Price | |
|---|---|---|---|---|
| 1 |
|
SEO ENGINEERED: Search Visibility Architecture for Web Apps at Scale (Quality Control for AI-Built... | $2.99 | Buy on Amazon |
| 2 |
|
Sitemap Handbook | $9.00 | Buy on Amazon |
| 3 |
|
Ultimate SEO: Grow Your Business Online | $4.99 | Buy on Amazon |
What robots.txt does—and does not do
robots.txt follows the Robots Exclusion Protocol defined by IETF RFC 9309. It publishes rules that compliant crawlers are requested to honor when accessing URI paths. The RFC is explicit: “These rules are not a form of access authorization.” [RFC 9309]
Where it belongs and how to use it
The file must be at the service’s top-level path as /robots.txt, use UTF-8, and be served as text/plain. Its rules are organized into user-agent groups with directives such as Disallow and Allow. Crawlers may also recognize a Sitemap record, for example:
Recommended Free Tools
#1 Best Overall
Sitemap: https://example.com/sitemap.xmlGoogle says it can find the sitemap the next time it crawls robots.txt. The sitemap record is an additional convention; it is not a substitute for the sitemap itself. [RFC 9309] [Google: Build and Submit a Sitemap]
Do not use it to hide sensitive content
A disallowed path can still be known or requested by people and other systems, and the robots file itself is public. Restrict confidential pages, files, or data with application- or server-level authentication and authorization. A crawler instruction is not a lock.
Keep the file reachable and changes deliberate
Under RFC 9309 behavior, when a crawler cannot retrieve robots.txt because of server or network errors, it assumes complete disallow under the specified conditions. The RFC also advises crawlers not to use a cached copy for more than 24 hours unless the file is unreachable. These protocol details make uptime and careful testing of edits important: an accidental outage or overly broad rule can affect crawling. [RFC 9309]
What a sitemap.xml is for
A sitemap gives search engines a structured list of URLs that a publisher considers important, along with supported metadata. Google supports formats in the Sitemaps protocol; XML is the most versatile, while a plain-text sitemap is simpler and limited to URLs for HTML and other indexable text. [Google: Build and Submit a Sitemap]
Quick wins for a faster PC:
Fix the driver behind crashes, sound loss and screen glitchesFind Drivers →Repair Windows errors before they cause bigger problemsFix Now →Rank #2
When a sitemap is useful
Google highlights large sites, new sites with few external links, and sites with rich media or news as cases where a sitemap can be especially useful. Its guidance says a sitemap may be unnecessary for a small site—about 500 relevant pages or fewer—if the site is comprehensively internally linked and has few relevant media or news files. That is an example with conditions, not a universal page-count cutoff. [Google: What Is a Sitemap]
Build it with accurate URLs and metadata
- List fully qualified absolute URLs that you want considered for Google Search, preferably canonical versions.
- Use UTF-8. A single sitemap can contain up to 50,000 URLs or 50 MB uncompressed; larger sites can split sitemaps and use a sitemap index. [Google: Build and Submit a Sitemap]
- Set
<lastmod>only when it is consistently and verifiably accurate. It should reflect a significant content, structured-data, or link change—not a trivial copyright-year update. - Do not spend time tuning
<priority>or<changefreq>for Google: it ignores those fields. [Google: Build and Submit a Sitemap]
A sitemap helps discovery, but does not order URLs, compel a crawl, or guarantee indexing. Google says it generally shows canonical URLs and will attempt to crawl URLs as listed; inclusion itself is not a promise of a search result. [Google: What Is a Sitemap]
Independent reader supportYour contribution helps us test, update, and keep practical guides available for everyone.What llms.txt is—and its limits
The llms.txt proposal describes a standard filename at a website’s root containing a concise, curated overview for LLM agents to consult when they need site material to help a user. Its intended role is different from a sitemap: the proposal argues that a sitemap may be too large for an LLM context, may not point to LLM-readable versions, and may omit useful external material. Those are the proposal’s rationale, not a guarantee that every model or AI product will find, read, or use the file. [The llms.txt proposal]
Does llms.txt help Google Search?
No. In its June 15, 2026 documentation update, Google said llms.txt files “aren’t needed for Google Search” and will not positively or negatively affect visibility or rankings. Google says a site may still maintain the file for other services or systems that use it. Do not expect it to improve Google rankings, produce AI citations, or generate traffic. [Google Search documentation updates]
Free tools Windows power users keep installed
One-click scans. No signup required.
Which files does your site need?
Use robots.txt when you need crawler access preferences
Publish it at the root when you have paths whose crawling you want to manage. Do not place private content behind a Disallow rule; use real access controls instead.
Use a sitemap when URL discovery warrants one
It is particularly useful for large or new sites, and sites with rich media or news. A small, well-linked site may not need one under Google’s stated conditions, but publishing a clean sitemap can still make the intended URL set explicit.
Consider llms.txt only for systems that use it
If you want to offer a concise, curated site guide to LLM tools that support the proposal, you can publish it independently. Treat adoption and outcomes as uncertain, and do not mistake it for a search-engine sitemap or access policy.
Many sites can use all three: robots.txt for crawler preferences, a sitemap for URL discovery, and llms.txt as an optional LLM-oriented overview. Their coexistence is useful precisely because none performs the other’s job.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
