Free tools Windows power users keep installed
One-click scans. No signup required.
iTechGuides is reader-supported. When you buy through links on our site, we may earn an affiliate commission. As an Amazon Associate I earn from qualifying purchases. Learn more
Adding llms.txt to a SaaS site does not guarantee Claude will find or read it. The file is a proposed Markdown guide to useful site content, not a crawler-permission setting, and available evidence does not show that it increases AI-search citations. One SaaS founder reported that Claude initially failed to fetch the file, then later found the company as a secondary reference after the team improved its discoverability. That account is a useful example of the gap between publishing a URL and an agent discovering it—not proof of how Claude generally behaves.
What llms.txt is meant to do
The llms.txt proposal describes a Markdown file placed at a site root or within a subpath, such as /docs/llms.txt. It acts as a concise map: brief context and guidance point an agent toward more detailed pages. The proposal describes its aim this way: “We propose adding a /llms.txt markdown file to websites to provide LLM-friendly content.”
![Claude AI for Beginners Bible: [5 in 1] The Ultimate Guide to Automate Your Work, Save Hours Every Week, and Use AI for Real-World Results](https://m.media-amazon.com/images/I/415+fSJacsL._SL160_.jpg)
The suggested structure starts with a site or project heading, may include a short blockquote and explanatory text, and can organize linked resources under section headings. A file in a subpath applies to URLs beneath that path; the proposal says agents should prefer the most specific applicable file. The intended workflow is to make relevant material easier to locate, not to copy an entire documentation site into one oversized file.
The Tool Desk
Outbyte PC Repair FREEClear out junk files and repair common Windows errorsFree Scan →Outbyte Driver Updater FREEScan for outdated or missing drivers - takes under a minuteDriver Scan →In its v2 proposal, llms.txt also recommends providing clean Markdown versions of relevant pages and identifying relationships with rel="alternate" for Markdown versions and rel="describedby" for the llms.txt file. These relationships can be published in HTML link elements or HTTP Link: headers. The proposal argues that HTML can be costly or difficult for agents to convert into clean text; that is its rationale, not proof that every agent will use these hints.
What the reported Claude test found
A founder at vacation-rental discovery SaaS CielStay described expecting this flow: “Agent reads llms.txt → calls /api/search → returns results.” In the founder’s account, Claude could not fetch the file during an initial “cold” attempt, even though the URL returned HTTP 200. The founder attributed the problem to the domain not appearing in search results, then described directing Claude to indexed third-party listings; Claude later found CielStay as a secondary reference.
The founder said the team then added llms.txt to its sitemap, linked it from a crawlable page, and added a <link> element. These details come from the founder’s first-person account, not an independent or controlled test. The post does not establish that indexing was the cause of the initial failure, that the changes caused the later discovery, or that Claude always needs a site to be indexed before it can fetch a URL.
What the wider evidence can—and cannot—tell you
A 2026 review of llms.txt usage reports an Ahrefs analysis of 137,210 domains in which 97% of the files received zero requests in May 2026. That figure is reported by llmstxt.studio; it describes requests observed in that analysis, not whether Claude or another model used a file internally without making a visible request.
PC Slower Than It Used to Be?
A free scan shows the junk files, broken settings and background clutter dragging Windows down - then fixes them in one click.Free scan · Windows 10 & 11Crashes, No Sound, or Screen Glitches?
Random freezes, missing sound and display glitches usually trace back to one bad driver. Find and replace yours safely.Free scan · under a minuteThe same review says that, as of its 2026 publication context, no provider had stated in writing that its search product reads llms.txt. It distinguishes signs such as Anthropic documentation hosting the file or Lighthouse checking for it from evidence that a consumer-facing AI answer actually fetched the file. Those signals do not establish live retrieval or a citation benefit.
Anthropic’s Claude 3.7 Sonnet system card says its general-purpose crawler follows robots.txt instructions when obtaining public web pages. That statement describes crawler policy; it does not explain how Claude’s interactive retrieval works or confirm support for llms.txt.
How llms.txt differs from robots.txt
These files have different jobs. The proposal describes robots.txt as communicating access rules to automated tools, while llms.txt provides contextual guidance and links for agents. An llms.txt file is not a permission grant, and publishing one does not replace robots.txt or override a crawler’s access rules.
How to make a useful file—and keep the test honest
Choose a scope and keep the index curated
Use a root-level file for a site-wide map, or a path-scoped file when a section such as documentation has its own relevant material. Link to the pages an agent would need to answer likely questions; an exhaustive dump is harder to scan and more burdensome to keep accurate. Check that linked pages are public when they are meant to be accessible, current, and clear about the product or API they describe.
Quick wins for a faster PC:
Scan for outdated or missing drivers - takes under a minuteDriver Scan →Repair Windows errors before they cause bigger problemsFix Now →Decide how pages will be maintained
Markdown links can lead to dedicated Markdown versions of pages, while HTML-only content can still be listed. The proposal’s recommended alternate and described-by relationships are implementation options, not a guarantee of use. A documentation platform that generates an index or Markdown pages can reduce manual publishing work, but automation does not establish that Claude or an AI search system will fetch them.
Best Value
Test discovery separately from direct access
A response of HTTP 200 confirms that a request to a URL succeeded; it does not show that an agent will discover the URL without being given it. To evaluate your setup, record the Claude product and mode, date, exact prompt, whether you supplied the llms.txt URL, any visible tool actions, the file’s HTTP response, and the result. Compare a cold-start prompt with a prompt that supplies the direct URL, and with one that points to ordinary crawlable pages. Keep these cases distinct: they test different routes to the content. Do not infer a general AI-search ranking effect from a single successful or failed attempt.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
