Quick wins for a faster PC:
Fix the driver behind crashes, sound loss and screen glitchesFind Drivers →Clear out junk files and repair common Windows errorsFree Scan →iTechGuides is reader-supported. When you buy through links on our site, we may earn an affiliate commission. As an Amazon Associate I earn from qualifying purchases. Learn more
llms.txt is a proposed Markdown index that helps AI agents navigate a website, especially its documentation; it is not a crawler directive or an established way to improve search visibility. Version 2 clarifies where files can live and how sites can point to Markdown pages. In an Ahrefs study of 137,210 traffic-active domains, 28% had a valid root file, but 97% of those files received no requests during May 2026.
What llms.txt is for
The proposal describes llms.txt as a compact guide to a site: a short introduction and curated links that help language models or other agents find relevant material. Documentation is a natural fit because it often consists of detailed pages that can be hard for an agent to locate or process efficiently.
It is not equivalent to robots.txt. It does not instruct crawlers to access or avoid pages, and publishing it does not itself make content eligible for search results or AI answers. The proposal is maintained by Jeremy Howard; its current page says it was first published September 3, 2024, and modified August 10, 2026. See the llms.txt v2 proposal.
What version 2 specifies
File location and scope
A file may appear at a site root or at a subpath. It covers URLs beneath its own path, and when more than one file applies, the most specific one takes precedence. So the format does not require every site to use only a root-level file.
#1 Best Overall
Required and recommended content
The only required section is an H1 naming the site or project. The proposal describes a short blockquote summary, optional explanatory text, and H2 sections containing lists of links to detailed pages. The summary is recommended, not mandatory; the H1 is the sole required section.
The links are meant to lead to useful, preferably machine-readable material. A section named ## Optional may mark secondary links an agent could skip when it needs less context. In v2, however, that label is a convention rather than a mechanical instruction that every parser must obey. The proposal’s v2 change notes explain the change.
Rank #2
Finding Markdown versions of pages
Where a Markdown version exists, v2 describes two URL patterns: append .md to the original path, as in page.html.md, or replace the original extension, as in page.md. A site can also identify these relationships for clients:
rel="alternate" type="text/markdown"identifies a Markdown alternative to a page.rel="describedby"identifies the llms.txt file that covers a page.
These relations can be published in HTML <link> elements or HTTP Link: headers. They provide a way to discover Markdown pages and the applicable index; they do not guarantee that a given agent will request or use them.
What is not part of v2
llms-full.txt is an ecosystem convention, not a filename defined by the v2 proposal. Older context-expansion companion-file behavior is also no longer part of the current proposal. A claim that the W3C has a Working Draft for llms.txt is inaccurate: the cited W3C item is an open repository issue, not a Working Draft.
What the 137,000-domain study found
Ahrefs published its study on June 15, 2026. It examined 137,210 domains in Ahrefs Web Analytics that received traffic in May 2026. The researchers checked each root for an HTTP 200 response at llms.txt, screened for Markdown rather than HTML or soft-error pages, and classified requests to llms.txt paths using Ahrefs Bot Analytics. The study did not assess whether files conformed to the specification.
| Finding | What it measures |
|---|---|
| 28% had a valid root file | Share of 137,210 traffic-active domains in Ahrefs’ panel; Ahrefs, 2026. |
| 97% of valid files had zero requests | Share of approximately 38,000 valid files with no requests during May 2026; Ahrefs, 2026. |
| 96% of requests were from bots | Share of requests to files that received traffic; Ahrefs, 2026. |
| 19.5% of requests were from named AI-tool categories | Share of requests to files with activity, combining AI agents, training crawlers, assistants, and retrieval bots; Ahrefs, 2026. It is not the share of all published files read by AI. |
| 1.1% of requests were from AI retrieval bots | Share of requests to files with activity; Ahrefs distinguishes these live-query crawlers from agent infrastructure, training crawlers, and assistants. |
| 12% of requests were from tools studying llms.txt | Share of requests to files with activity, including SEO/GEO/AEO auditing, discovery, checking, and research tools; Ahrefs, 2026. |
The 28% result is not a web-wide adoption rate. Ahrefs says its customers skew more technical and SEO-aware than the web overall, so this panel should be treated as an upper-bound-like result rather than a representative estimate. The request breakdowns also apply only to files that received activity, not to all sites or all published files. This is a one-month observation from one vendor panel: it cannot establish future usage or whether publishing a file causes a search or citation benefit. Read Ahrefs’ study and methodology.
Recommended Free Tools
Does llms.txt work for search visibility?
The available evidence does not establish llms.txt as a search-ranking or AI-citation lever. Ahrefs reports that Google’s generative-AI guidance says machine-readable files such as llms.txt are not needed to appear in generative AI search. It also reports that Google Search Advocate John Mueller described the file as “not done for search,” while noting a possible temporary use for saving tokens when AI coding tools parse developer documentation. That wording is reported by Ahrefs in its June 15, 2026 study; it should not be read as proof that no agent will ever use the format.
Best Value
- Core i3-8100 3.6GHz 4-Core Processor
- 8GB (1x 8GB) DDR4 Memory
- 480GB SATA 6Gbps SSD
- 2x Integrated 1GbE RJ45 NIC Ports -- Bring Your Own PCIe NIC
- Shallow/Short Depth Rackmount Server
The narrower conclusion is that the file is a proposed navigation aid, not an established general search tactic. The May 2026 Ahrefs results show that most valid files in its sample went unrequested during that month, while some files did receive requests from bots, including AI-tool categories.
Independent reader supportYour contribution helps us test, update, and keep practical guides available for everyone.Should your site publish one?
For a documentation-heavy site, a concise, maintained index may be worthwhile as a low-cost navigation aid—particularly when useful Markdown documentation already exists. For a site focused on search visibility, the evidence here does not justify prioritizing llms.txt as a proven ranking or citation tactic.
Before adding one, weigh the practical fit rather than assuming a visibility payoff:
The Tool Desk
Outbyte Driver Updater FREEFix the driver behind crashes, sound loss and screen glitchesFind Drivers →Outbyte PC Repair FREERepair Windows errors before they cause bigger problemsFix Now →- Does the site have substantial documentation or structured content that agents might need to navigate?
- Can the team keep a manually curated file accurate, or generate it alongside existing Markdown documentation without creating stale links?
- Are Markdown page versions already served, and do their URLs work as declared?
- Can the team expose and maintain the recommended discovery relations?
- Is upkeep more valuable than other documentation work the team could do?
Documentation platforms such as Mintlify or GitBook may be relevant when a team is already choosing documentation infrastructure; their relevance does not establish that buying a platform improves AI visibility.
Quick Recap
How to make an llms.txt useful
- Put it where its scope makes sense. Use a root file for site-wide documentation or a subpath file for a distinct section; the applicable file covers URLs beneath its location.
- Identify the project and orient the reader. Include the required H1 and a concise summary that explains what the linked material contains.
- Organize links by task or subject. Use H2 headings and curated Markdown links to the most useful detailed pages rather than treating the file as a duplicate of the whole site.
- Prefer useful Markdown destinations where available. Use a consistent supported URL pattern, then expose the Markdown alternative and applicable llms.txt through the proposed discovery relations if the site can keep them accurate.
- Maintain and validate it. Remove stale links as documentation changes; a compact index is only useful if its destinations remain relevant and accessible.
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
