What’s actually slowing this PC down?
Pick the symptom - the matching free tool is one click away.
To find a website’s llms.txt, first inspect the page HTML and its HTTP Link response header for rel="describedby". If the site does not advertise a location, test the most specific path that could govern the page, then the origin root, and finally documentation subdomains or subpaths. A root-level 404 only proves that /llms.txt is absent; it does not rule out a file under /docs/ or another scoped directory.
The reliable discovery sequence
- Inspect the page itself. Look for a link relation that identifies an llms.txt file.
- Try the closest applicable path. A file can apply to every URL beneath its directory.
- Check the origin root. Request
https://host.example/llms.txteven when a deeper path was tried. - Check documentation locations. Software publishers may place the file on a docs subdomain or under a documentation path.
- Use public directories only as leads. Confirm every listing against the live site.
- Read the file as a map. Follow its curated links to the documentation or content you actually need.
1. Inspect the page HTML and headers first
The proposal for llms.txt recommends discovery metadata. In the page source, search for a <link> element whose relation is describedby. The linked URL is the most direct answer for that page’s scope. The same relation can be sent in an HTTP Link response header.
Using curl
curl -sS -D - https://example.com/docs/api/auth -o page.html
grep -iE 'llms|describedby|alternate' page.htmlReview the response headers printed by -D -. A header may look like Link: <https://example.com/docs/llms.txt>; rel="describedby". In the saved HTML, also search for rel="describedby". Do not confuse rel="alternate" type="text/markdown" with llms.txt discovery: that relation advertises a Markdown representation of the individual page, not a site or path map.
Using browser developer tools
- Open the target page.
- Choose View page source or open Developer Tools and select the Elements panel.
- Search for
describedby,llms.txt, andalternate. - In the Network panel, reload the page, select the document request, and inspect Response Headers for
Link.
A declared relation is preferable to guessing because it tells you which map is intended for that page. If no relation appears, continue with path checks.
Windows Errors? Fix Them Before They Spread
Repair common Windows errors and clear accumulated junk for a smoother, more stable PC - no reinstall needed.Free scan · no reinstallOutdated Drivers Are Slowing You Down
One free scan finds every outdated or missing driver and matches the right update for your exact hardware.Free scan · exact hardware match#1 Best Overall
- Performance: Advanced read speeds of up to 130MB/s for everyday data storage & transfers²
- Speed: Transfer speeds up to 10x faster than standard USB 2.0 flash drives²
- Durability: Sturdy, light-weight design with convenient and modern sliding collar cap design protects content when not in use
- Reliability: Essential mobile storage solution ideal for transferring large files such as movies, videos, photos, music & documents
- Compatibility: Compatible with most Type-A USB 3.2 Gen 1/USB 3.0 PC and Mac laptop and desktop computers, backwards compatible with USB 2.0
2. Test the most specific path before the root
The proposal allows an llms.txt file to be published at a path and to cover URLs beneath that path. For https://example.com/docs/api/auth, test these candidates in order:
https://example.com/docs/api/llms.txthttps://example.com/docs/llms.txthttps://example.com/llms.txt
The first existing file is not automatically the only one. If several respond successfully, use the most specific file for the page you are researching, because its scope is narrower. A path-level map may describe API documentation while the root map covers the entire site.
Check status and content without downloading a large response
for u in
https://example.com/docs/api/llms.txt
https://example.com/docs/llms.txt
https://example.com/llms.txt
do
printf '%s: ' "$u"
curl -sS -o /dev/null -w '%{http_code} %{content_type}n' "$u"
doneA 200 response with a text content type is a strong candidate. Follow redirects, but record the final URL; a redirect can move the file to a different host or path. A 403, authentication challenge, or server error is not proof that no file exists.
3. Check documentation subdomains and subpaths
Documentation-heavy services commonly separate product content from their marketing site. If the page is on docs.example.com, test https://docs.example.com/llms.txt. If the documentation is at example.com/docs/, test https://example.com/docs/llms.txt even when the origin root returns 404. Also inspect the documentation page’s own HTML and headers; a declared relation can point to a non-obvious location.
Free tools Windows power users keep installed
One-click scans. No signup required.
Rank #2
- Transfer speeds up to 10x faster than standard USB 2.0 drives (4MB/s); up to 130MB/s read speed; USB 3.0 port required. Based on internal testing; performance may be lower depending upon host device. 1MB=1,000,000 bytes
- Backward compatible with USB 2.0
- Secure file encryption and password protection(2)
4. Use community directories for broad discovery
When you want to find many publishing sites rather than one known site, directories can supply candidates. The practical guide names directory.llmstxt.cloud and llmstxt.site. Community references also link to these indexes through awesome-answer-engine-optimization.
Directory entries can become stale, change location, or omit a path-scoped file. Open the listed URL, check its current HTTP response, and compare its scope with the page you care about. Treat a directory as a search index, never as authoritative proof that the file is still published.
5. Read llms.txt as a navigation map
The format described at llmstxt.org is Markdown. A typical file contains:
- An H1 naming the site or covered project.
- An optional blockquote summarizing the site.
- Optional explanatory prose.
- H2 sections containing links to detailed resources.
The file is not normally the complete documentation. Follow the links under the relevant section, and check whether the URLs remain within the path that published the map. A path-scoped file should be interpreted in relation to the pages beneath that path, not assumed to describe every host owned by the organization.
Rank #3
- 256GB ultra fast USB 3.1 flash drive with high-speed transmission; read speeds up to 130MB/s
- Store videos, photos, and songs; 256 GB capacity = 64,000 12MP photos or 978 minutes 1080P video recording
- Note: Actual storage capacity shown by a device's OS may be less than the capacity indicated on the product label due to different measurement standards. The available storage capacity is higher than 230GB.
- 15x faster than USB 2.0 drives; USB 3.1 Gen 1 / USB 3.0 port required on host devices to achieve optimal read/write speed; Backwards compatible with USB 2.0 host devices at lower speed. Read speed up to 130MB/s and write speed up to 30MB/s are based on internal tests conducted under controlled conditions , Actual read/write speeds also vary depending on devices used, transfer files size, types and other factors
- Stylish appearance,retractable, telescopic design with key hole
Automate the checks
Python: inspect metadata and candidate paths
import re
from urllib.parse import urljoin, urlparse
import requests
page_url = "https://example.com/docs/api/auth"
headers = {"User-Agent": "llms-txt-discovery/1.0"}
r = requests.get(page_url, headers=headers, timeout=20)
print("page:", r.status_code, r.url)
for value in r.headers.get("Link", "").split(","):
if "describedby" in value.lower():
print("Link header:", value.strip())
for match in re.finditer(r']*>', r.text, re.I):
tag = match.group(0)
if re.search(r'rel=["']?[^"'>]*describedby', tag, re.I):
print("HTML relation:", tag)
parsed = urlparse(r.url)
parts = [p for p in parsed.path.split("/") if p]
candidates = []
for end in range(len(parts), -1, -1):
prefix = "/" + "/".join(parts[:end]) + "/"
candidates.append(f"{parsed.scheme}://{parsed.netloc}{prefix}llms.txt")
candidates.append(f"{parsed.scheme}://{parsed.netloc}/llms.txt")
seen = set()
for candidate in candidates:
if candidate in seen:
continue
seen.add(candidate)
x = requests.get(candidate, headers=headers, timeout=20, allow_redirects=True)
print(candidate, "->", x.status_code, x.url, x.headers.get("Content-Type", "")) This script reports metadata, then walks from the page’s directory toward the origin. For production use, add retry and rate-limit handling and cache responses so you do not repeatedly request the same host.
Node.js: fetch headers and candidates
const page = new URL('https://example.com/docs/api/auth');
const response = await fetch(page, { redirect: 'follow' });
console.log('page:', response.status, response.url);
console.log('Link:', response.headers.get('link'));
const html = await response.text();
for (const tag of html.matchAll(/<link\b[^>]*>/gi)) {
if (/rel=["']?[^"'>]*describedby/i.test(tag[0])) console.log('HTML:', tag[0]);
}
const finalPage = new URL(response.url);
const segments = finalPage.pathname.split('/').filter(Boolean);
const candidates = [];
for (let i = segments.length; i >= 0; i--) {
const prefix = '/' + segments.slice(0, i).join('/') + '/';
candidates.push(new URL(prefix + 'llms.txt', finalPage.origin));
}
for (const candidate of new Set(candidates.map(String))) {
const r = await fetch(candidate, { method: 'HEAD', redirect: 'follow' });
console.log(candidate, '->', r.status, r.url, r.headers.get('content-type'));
}How to verify a candidate file
- Confirm the final URL. Record redirects and ensure the host is one you trust.
- Check the response. Look at status, content type, and whether the body is readable Markdown.
- Check scope. A file under
/docs/api/should be treated as applying to that subtree unless the publisher explains otherwise. - Check freshness. Note
Last-ModifiedorETagwhen available, and avoid assuming that a directory listing is current. - Validate links. Resolve relative links against the llms.txt URL and test important destinations.
What a missing file means
llms.txt is an optional community convention, not a mandatory web resource. Chrome for Developers states: “If the file is not provided by the server (resulting in a 404), the audit is marked as Not Applicable (N/A), as providing the file is optional at the moment.” See the Lighthouse llms.txt documentation, last updated May 5, 2026.
Do not equate llms.txt with robots.txt. The proposal describes robots.txt as communicating crawler access preferences, while llms.txt is an on-demand orientation document. Publishing a file also does not establish that a particular AI product will fetch or honor it. The community references at AI Discovery Standards classify the idea as a convention and do not document a universal engine commitment.
Choosing a discovery method
| Method | Coverage | Directness | Scope and freshness |
|---|---|---|---|
| HTML or HTTP relation | One known page | Highest; publisher declares the map | Usually the page’s intended scope; live response |
| Path and root requests | One known host | Good, but relies on conventional locations | Can reveal deeper scoped files; live response |
| Community directory | Many candidate sites | Indirect; listing is a lead | Broad coverage, but entries may lag |
Troubleshooting common failures
You receive 404 from /llms.txt
Try the page’s nearest directory and documentation path. The 404 establishes only that the root file is missing.
Quick wins for a faster PC:
Repair Windows errors before they cause bigger problemsFix Now →Scan for outdated or missing drivers - takes under a minuteDriver Scan →Clear out junk files and repair common Windows errorsFree Scan →Rank #4
- High-speed USB 3.0 performance of up to 150MB/s(1) [(1) Write to drive up to 15x faster than standard USB 2.0 drives (4MB/s); varies by drive capacity. Up to 150MB/s read speed. USB 3.0 port required. Based on internal testing; performance may be lower depending on host device, usage conditions, and other factors; 1MB=1,000,000 bytes]
- Transfer a full-length movie in less than 30 seconds(2) [(2) Based on 1.2GB MPEG-4 video transfer with USB 3.0 host device. Results may vary based on host device, file attributes and other factors]
- Transfer to drive up to 15 times faster than standard USB 2.0 drives(1)
- Sleek, durable metal casing
- Easy-to-use password protection for your private files(3) [(3)Password protection uses 128-bit AES encryption and is supported by Windows 7, Windows 8, Windows 10, and Mac OS X v10.9+; Software download required for Mac, visit the SanDisk SecureAccess support page]
The page source has no relation
Continue with scoped-path and root checks. The proposal recommends relations but does not require every publisher to emit them.
The server returns HTML instead of Markdown
Check for a login page, single-page-app fallback, or custom error page. Inspect the body and final URL; a status of 200 alone is insufficient.
A directory link is dead
Search the target host’s current HTML and headers, then test its docs subdomain or path. Remove the stale directory result from your working list.
Requests are blocked or rate-limited
Slow down, identify your client honestly, honor the site’s access policy, and retry later. Do not treat a temporary 403 or 429 as evidence that no file exists.
Recommended Free Tools
Best Value
- USB-C 2-in-1 storage OTG: The Lexar JumpDrive Dual Drive D40E features USB Type-A and Type-C connectors in a slim, portable form factor for easy device compatibility
- Transfer speeds up to 100MB/s: Based on internal testing, performance may vary depending upon the host device, interface, and usage conditions. 1MB=1,000,000 bytes
- Plug and Play: Widely compatible with USB Type-C smartphones, tablets, laptops, Macs, and traditional Type-A devices, no software installation required. The 360° swivel design allows for easy switching between connectors without the hassle of losing a cap
- Durable & Compact: The Lexar D40E USB memory stick features a metal enclosure, withstands temperatures from 0° to 50° C (32°F to 122°F), and is lightweight at 26g with dimensions of 70.4 x 16.9 x 11.7mm
- Security & Warranty: Securely protects files using an advanced security software solution with 256-bit AES encryption. Backed by a Lexar 3-year limited warranty
Several files exist
Prefer the file with the narrowest path that contains the page. Keep the root file as a broader index when you need site-wide orientation.
Independent reader supportYour contribution helps us test, update, and keep practical guides available for everyone.Or skip the browser setup
If you need a rendered image of a page while documenting your discovery work, ScreenshotNeo provides a website screenshot API and MCP server. It accepts a URL in one request and can return PNG, JPEG, WebP, or PDF. Cookie banners, newsletter popups, and chat widgets are removed before the shot; bot checks, blank pages, timeouts, failed loads, and cache hits are not billed, and response headers identify the page verdict and billing status. Its MCP server exposes take_screenshot, get_page_info, and capture_pdf for Claude, Cursor, and other MCP clients.
For API parameters and all 63 options, see the ScreenshotNeo documentation.
cURL
curl -G "https://api.screenshotneo.com/v1/shot" -d access_key=YOUR_API_KEY --data-urlencode url=https://llmstxt.org/ -o shot.webpPython
import requests
r = requests.get("https://api.screenshotneo.com/v1/shot", params={"access_key": "YOUR_API_KEY", "url": "https://llmstxt.org/"}, timeout=90)
open("shot.webp", "wb").write(r.content)Node.js
const q = new URLSearchParams({ access_key: 'YOUR_API_KEY', url: 'https://llmstxt.org/' });
const res = await fetch(`https://api.screenshotneo.com/v1/shot?${q}`);The Free plan includes 1,000 screenshots per month with no card. Paid plans start at $5 for 3,000 shots; every feature is available on every plan. Create a free ScreenshotNeo account.
Do these 3 things before closing this tab:
1Scan for outdated or missing drivers - takes under a minute2Repair Windows errors before they cause bigger problems3Fix the driver behind crashes, sound loss and screen glitchesFrequently Asked Questions
Does a file have to be named exactly llms.txt?
The convention and proposal discussed here use the literal filename llms.txt. A publisher may expose an equivalent document through another URL, but you should not label it an llms.txt file without checking the publisher’s declaration.
Can one website publish more than one llms.txt file?
Yes. The proposal permits path-scoped files, so a root map and one or more documentation maps can coexist. Select the narrowest applicable path for the page you are researching.
Will finding llms.txt change how an AI crawler behaves?
Not necessarily. Publication is an orientation signal, not a universal enforcement mechanism; individual AI systems decide whether to request or follow it.
Should I cache a directory result?
Cache it as a lead, not as proof. Revalidate the listed URL against the live host before relying on its links or scope.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
