Quick wins for a faster PC:
Repair Windows errors before they cause bigger problemsFix Now →Fix the driver behind crashes, sound loss and screen glitchesFind Drivers →iTechGuides is reader-supported. When you buy through links on our site, we may earn an affiliate commission. As an Amazon Associate I earn from qualifying purchases. Learn more
llms.txt is a proposed Markdown file that gives compatible AI agents a concise guide to a website and links to selected resources. It is not a crawler-permission file, and publishing it does not guarantee that an AI answer engine will fetch or use it. Google Search says it does not treat the file as special markup, including for its generative AI features.
What llms.txt is—and what it is not
The proposal describes llms.txt as a publisher-controlled resource map: a short explanation of a site, organized links to useful pages, and, where available, links to clean Markdown versions. The aim is to make relevant information easier for compatible agents to find and interpret, especially on sites whose pages are designed primarily for human readers.
It is not a replacement for the pages themselves. Nor is it equivalent to robots.txt: the proposal assigns the two different roles. An llms.txt file points agents toward useful resources; crawl preferences belong in the mechanisms intended for that purpose.
What’s actually slowing this PC down?
Pick the symptom - the matching free tool is one click away.
The proposal was authored by Jeremy Howard and first published on September 3, 2024. Its v2 page was modified on August 10, 2026. Those dates describe the proposal, not a universal standard adopted by AI providers. Read the llms.txt proposal.
Do AI answer engines read llms.txt?
Google Search and AI features
Google Search Central says Google Search does not use llms.txt as special markup, including for generative AI capabilities. Its guidance says maintaining the file neither helps nor harms a site’s Search visibility or rankings. Google clarified in its June 2026 Search Central update log that the file is not needed for Google Search, though site owners may maintain one for other systems that use it. That position is specific to Google; it should not be treated as a policy statement for every provider. Google Search Central’s guide to AI features and Search Central update log.
Other providers
The proposal describes an intended workflow in which compatible agents read or search the file, follow relevant links, and fetch pages for details. That describes the proposal’s purpose, not proof of implementation. A company publishing an llms.txt file for its own documentation does not establish that its answer product fetches such files from arbitrary websites. The cited official material does not establish systematic use by ChatGPT, Claude, Gemini, or Perplexity.
So the practical answer is conditional: an agent can use the file if its systems choose to fetch and follow it, but you should not assume that a particular answer engine does so without provider-specific evidence.
What belongs in an llms.txt file?
The proposal specifies this order. The only required section is the H1 naming the site or project; the other parts are optional.
- Optional byte-order mark. Include one only if needed by the file’s encoding workflow.
- Required H1. Name the site or project, for example
# Acme Documentation. - Optional blockquote. Briefly explain what the site covers and whom it serves.
- Optional prose. Add concise context that helps interpret the links, without introducing additional headings in this portion.
- Optional H2 sections and link lists. Group selected pages under useful headings such as
## Getting started,## API reference, or## Policies. Use Markdown links with short, informative descriptions.
Choose useful entry points rather than treating the file as a complete sitemap. Prefer canonical URLs. If your site offers clean Markdown counterparts, the proposal suggests exposing them by appending or substituting .md in the URL, so an agent can access a less presentation-heavy version of a page.
How to create and publish one
- Decide what the file covers. Put it at the site root for broad coverage, or at a more specific path such as
/docs/llms.txtfor a documentation section. The proposal says a file covers URLs beneath its path and that the most specific applicable file should be used. - Write the file in the proposed order. Start with the site-name H1, then add the optional context and carefully grouped link sections. Keep descriptions clear and concise; do not use the file to make unsupported claims about how providers will treat the site.
- Link to relevant, accessible pages. Select authoritative resources that answer likely questions. When clean Markdown counterparts exist, include those where they are useful.
- Advertise the file and Markdown alternatives. The proposal recommends pointing to the applicable file with
rel="describedby". For Markdown page versions, it recommendsrel="alternate"withtype="text/markdown". These relations can be conveyed in HTML link elements or HTTPLinkresponse headers. Follow the proposal’s examples for the syntax that fits your server setup. See the proposal’s discovery guidance. - Keep it aligned with the site. Check that the file is publicly accessible, its descriptions remain accurate, and its links still lead to useful pages after content or URL changes.
Which implementation approach fits your site?
| Site setup | Possible approach named in the proposal | What to check |
|---|---|---|
| WordPress | Yoast SEO and AIOSEO are named as plugins with llms.txt generation functionality. | Confirm the current feature set, how output is maintained, and whether it supports the structure and scope your site needs. |
| Hosted documentation platform | Mintlify is named as generating llms.txt and Markdown page versions; GitBook is named as serving llms.txt for published documentation sites. | Check current support for path-scoped files, Markdown counterparts, and discovery links. |
| Custom site or build system | The proposal lists code and platform integrations, including JavaScript and documentation plugins; a team could also generate the file from its content source or build process. | Decide how curated links will stay current and who owns maintenance. |
These are implementation paths mentioned by the proposal, not independent product comparisons. The cited material does not establish current prices or feature parity. For any approach, compare how well it fits your existing publishing workflow, whether it can produce the scoped files and Markdown versions you want, and how reliably it will stay up to date. The proposal’s examples and integrations.
Quick Recap
Best Value
Rank #4
Common mistakes to avoid
- Promising Google benefits. Google says the file does not improve or harm its Search visibility or rankings.
- Assuming every engine supports it. The proposal’s intended use and a provider’s own documentation file are not evidence that its answer engine reads the file for other sites.
- Using it as a crawler-control file. It is a resource map, not a substitute for crawl-control mechanisms.
- Listing every URL. A focused selection of useful pages is more consistent with the proposal’s purpose than an undifferentiated URL dump.
- Letting links go stale. Incorrect descriptions, moved pages, or inaccessible URLs undermine the file’s usefulness to any agent that does consult it.
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
