Quick wins for a faster PC:
Fix the driver behind crashes, sound loss and screen glitchesFind Drivers →Clear out junk files and repair common Windows errorsFree Scan →iTechGuides is reader-supported. When you buy through links on our site, we may earn an affiliate commission. As an Amazon Associate I earn from qualifying purchases. Learn more
To download a website sitemap, first check https://example.com/robots.txt for a Sitemap: line. Open the listed URL, then save the response in your browser or retrieve it with curl or wget. If the response is a sitemap index, download the child sitemaps listed in its <loc> elements.
What a sitemap download gives you
A sitemap is an XML document that lists URLs a site wants search engines to discover. A regular URL sitemap uses a <urlset> root and contains page entries. A sitemap index uses a <sitemapindex> root and contains links to other sitemap files. The filename is only a convention: inspect the response body rather than assuming that every sitemap is named sitemap.xml.
Downloading a sitemap lets you inspect URLs locally, feed them into an auditing script, archive a copy, or retrieve the individual files referenced by an index. It does not guarantee that a search engine will crawl or index those URLs. Google describes sitemap submission as a hint, not a guarantee.
The Tool Desk
Outbyte PC Repair FREERepair Windows errors before they cause bigger problemsFix Now →Outbyte Driver Updater FREEFix the driver behind crashes, sound loss and screen glitchesFind Drivers →Find the sitemap URL
Check robots.txt first
- Replace
example.comwith the site’s exact host and openhttps://example.com/robots.txt. - Search for one or more lines beginning with
Sitemap:, such asSitemap: https://www.example.com/sitemap.xml. - Copy the complete URL after the colon, including its protocol, path, capitalization, and any filename suffix.
The Sitemap protocol allows site owners to advertise sitemap locations in robots.txt. A site can publish more than one sitemap, so record every Sitemap: line.
#1 Best Overall
Try common paths when robots.txt has no entry
If no declaration appears, try these addresses on the same domain:
https://example.com/sitemap.xmlhttps://example.com/sitemap_index.xml
These are discovery guesses, not guaranteed locations. A content-management system may generate a different path or document it in its own help pages. Also try the site’s canonical HTTPS host if the HTTP version redirects.
Download the sitemap in a browser
- Paste the discovered sitemap URL into the address bar.
- Confirm that the response loads and that its content resembles XML rather than an HTML error page.
- Use the browser’s save or download command. Depending on the browser, right-click the page and choose Save as, or use the main menu’s save command.
- Choose a local filename ending in
.xmland retain the original file contents.
A browser is the simplest option for a one-off check because it immediately shows whether the URL responds. If the browser displays formatted XML, saving the page still preserves the underlying document; do not copy only the visibly collapsed portion of a large tree.
Crashes, No Sound, or Screen Glitches?
Random freezes, missing sound and display glitches usually trace back to one bad driver. Find and replace yours safely.Free scan · under a minuteWindows Errors? Fix Them Before They Spread
Repair common Windows errors and clear accumulated junk for a smoother, more stable PC - no reinstall needed.Free scan · no reinstallDownload with curl or wget
curl
Use the exact URL you discovered and follow redirects:
Rank #2
- HTML CSS Design and Build Web Sites
- Comes with secure packaging
- It can be a gift option
curl -L -o sitemap.xml 'https://www.example.com/sitemap.xml'-L follows HTTP redirects and -o writes the response to the named file. Keep the URL in quotes so shell characters cannot alter it.
wget
wget -O sitemap.xml 'https://www.example.com/sitemap.xml'-O selects the output filename. For an investigation, you can also save response headers with curl:
curl -L -D sitemap-headers.txt -o sitemap.xml 'https://www.example.com/sitemap.xml'Check the status and headers when a download looks suspicious. A file saved successfully can still be an error document returned with an HTTP success status.
Verify that the file is really a sitemap
- Open the downloaded file in a text editor or XML-aware viewer.
- Look near the beginning for a
<urlset>or<sitemapindex>root element. - For a URL sitemap, inspect
<url>entries and their<loc>values. - For an index, inspect each child
<sitemap>and its<loc>value.
If you see a login page, cookie-consent HTML, a “not found” message, or a server error instead, you did not retrieve a usable sitemap. Validate the URL and response before processing it with a parser.
Rank #3
Download every file in a sitemap index
An index is a directory of sitemap URLs, not the page list itself. Follow each child <loc> and save the responses separately.
- Download the index, for example with
curl -L -o sitemap-index.xml 'https://www.example.com/sitemap_index.xml'. - Extract the child URLs from its
<loc>elements. - Request each child URL and give each file a distinct name, such as
sitemap-posts.xmlorsitemap-products.xml. - Inspect each child as a
<urlset>or, on a very large site, as another index.
Do not treat the index URL as the complete inventory. The page URLs are in the child files.
Simple shell extraction
For a trusted index whose XML uses one URL per line, this shell pipeline extracts locations and downloads them:
What’s actually slowing this PC down?
Pick the symptom - the matching free tool is one click away.
grep -oP '(?<=<loc>).*?(?=</loc>)' sitemap-index.xml | while IFS= read -r url; do
name=$(printf '%s' "$url" | sed 's#[^A-Za-z0-9._-]#_#g')
curl -L -o "$name" "$url"
doneFor production processing, use an XML parser rather than regular expressions. XML namespaces, line wrapping, escaping, and unusual formatting can make text matching unreliable.
Limits and large-site handling
Google’s published limit is 50 MB uncompressed or 50,000 URLs per sitemap file. Sites exceeding either limit split their URLs across multiple sitemap files and commonly expose those files through an index. Compression can reduce transfer size, but the stated size limit is uncompressed, so do not use a compressed download to bypass it.
When handling a large index, download child files one at a time or with controlled concurrency, retain the original filenames, and record HTTP status and retrieval time. This makes it easier to identify one failed child without repeating the entire job.
Browser versus command line
| Method | Best for | Strength | Watch for |
|---|---|---|---|
| Browser | One-off inspection | Immediate visual confirmation that the URL loads | Saving a rendered view instead of the complete response |
| curl | Repeatable downloads and scripts | Exact output filename, redirect handling, and header capture | Assuming a successful command means valid XML |
| wget | Simple terminal retrieval | Short command with explicit output file | Not checking status or response contents |
| Crawler software | Broad site audits and many related files | Can combine sitemap data with link and status analysis | Unnecessary complexity for downloading one sitemap |
Troubleshoot failed downloads
The URL returns 404 or another HTTP error
Recheck the domain, protocol, spelling, and path. Copy the URL again from robots.txt, try the canonical HTTPS host, and open it in a browser. A guessed path may simply not exist.
You downloaded HTML instead of XML
Inspect the first lines and the response headers. Common causes include a custom error page, a login requirement, a redirect to a home page, or a bot-protection page. Use the exact sitemap URL, follow redirects, and resolve any access requirement with the site owner rather than parsing the HTML as XML.
Best Value
- Brand: Wiley
- Set of 2 Volumes
- A handy two-book set that uniquely combines related technologies Highly visual format and accessible language makes these books highly effective learning tools Perfect for beginning web designers and front-end developers
The browser shows nothing or times out
Retry after checking the host and network connection. Test the URL with curl so you can see whether the server returns an HTTP status. A temporary server failure is different from a nonexistent sitemap path.
Google cannot process the sitemap
Use Search Console’s Sitemaps report to view processing errors for sitemaps submitted through that report or its API. Sitemaps discovered through other methods may not appear there. First confirm that the downloaded response is valid XML and that every <loc> contains a complete URL.
The file is too large
Look for a sitemap index and process its child files. A single Google sitemap file is limited to 50,000 URLs or 50 MB uncompressed, so a larger collection must be split.
If your goal is a visual record of how a sitemap page renders rather than the raw XML download, ScreenshotNeo can capture the URL through one HTTP request. It is not a replacement for downloading XML for parsing, but it is useful when you need a screenshot of a sitemap response or an error page. See the ScreenshotNeo documentation for request options. Before capture it accepts cookie or consent banners and removes more than 60 known consent platforms, newsletter popups, and chat widgets; each cleanup step can be disabled. Bot checks, CAPTCHAs, blank pages, timeouts, failed loads, and cache hits are not billed, and the response identifies the page verdict and billing status in headers. Its MCP server provides Yes. If the sitemap URL is publicly accessible, you can open or request it like any other public URL. Access controls or authentication may prevent retrieval. Download each declared URL. They may be separate URL sitemaps or indexes for different content types. Use whichever URL the site declares. Those filenames are common conventions, not requirements. Not necessarily. It contains URLs the site has chosen to publish in that sitemap, and the file may omit pages or be stale. Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.Or skip the browser setup
curl -G "https://api.screenshotneo.com/v1/shot" -d access_key=YOUR_API_KEY --data-urlencode url=https://example.com/sitemap.xml -o shot.webptake_screenshot, get_page_info, and capture_pdf tools for Claude, Cursor, and other MCP clients. The Free plan includes 1,000 screenshots per month with no card, and paid plans start at $5 for 3,000 shots. Create a free ScreenshotNeo account.Use the downloaded sitemap responsibly
<lastmod> as publisher-supplied metadata, not proof that a page changed on that date.Frequently Asked Questions
Can I download a sitemap without owning the website?
What if a site has several Sitemap lines?
Should I use sitemap.xml or sitemap_index.xml?
Does a sitemap contain every page on a website?
Quick Recap
