What’s actually slowing this PC down?
Pick the symptom - the matching free tool is one click away.
To generate a sitemap by scraping a website, crawl its reachable internal links, then filter the results down to unique, preferred canonical pages that should appear in search. Write those URLs to XML or a plain-text sitemap, validate the file, publish it, and reference it in robots.txt or submit it through Google Search Console. First check whether your CMS or site software already creates a sitemap: Google says that is the best option when available.
Check whether your site already has a sitemap
For a site you own or administer, start with the CMS or website software rather than building a crawler. Check its documentation and common sitemap locations, and inspect the site’s robots.txt for a Sitemap: line. If the site’s software maintains a sitemap, it is usually a better source of truth than a separate crawl-derived file. Google’s sitemap-building guidance recommends having website software generate the file when possible.
A crawl is useful when you need an inventory of reachable pages, when your platform does not provide a suitable sitemap, or when you want to audit what internal links expose. Treat the crawl as a set of candidates, not as a ready-to-publish sitemap.
Choose a crawl method and define its scope
Use a desktop crawler for a visual workflow
- Enter the site’s starting URL in Screaming Frog SEO Spider and run the crawl.
- When the crawl finishes, choose Sitemaps > XML Sitemap to create an XML file.
- Review the generated entries and remove any URLs that do not belong in the site’s search-facing sitemap before publishing.
Screaming Frog documents this crawl-to-sitemap workflow in its XML sitemap tutorial. The vendor says its free Lite edition supports crawls of up to 500 URLs; check the product’s current limits before relying on that figure.
Do these 3 things before closing this tab:
1Clear out junk files and repair common Windows errors2Fix the driver behind crashes, sound loss and screen glitches3Repair Windows errors before they cause bigger problems#1 Best Overall
Use Scrapy when you need repeatable crawl logic
In a coded workflow, define a valid seed URL and an explicit allowed host or domain scope, then have the spider parse each response and follow eligible internal links. Scrapy spiders start with URLs and can yield follow-up requests from callbacks; its documentation describes allowed-domain constraints. See the Scrapy spider documentation for the framework’s spider model.
Set crawl boundaries before collecting URLs: decide whether subdomains are in scope, which protocols and hostnames are preferred, and which paths should be skipped. Crawl only sites you are authorized to access, and use a sensible request rate. A sitemap crawler is not a way to bypass access controls or a site’s policies.
Turn crawled URLs into sitemap entries
Normalize and assess every candidate before writing the file. The goal is not to list everything the crawler discovered; it is to list the preferred URLs for useful pages you want search engines to discover.
- Resolve equivalent addresses: choose a consistent preferred scheme and host, such as HTTPS and either the www or non-www hostname. Where equivalent content has multiple URLs, keep the canonical URL rather than every variant; Google’s guidance is to choose the canonical URL.
- Remove URL noise: remove fragments and strip tracking or session parameters when they do not identify a distinct page. Apply the same normalization rules throughout the crawl so duplicate variants collapse to one URL.
- Check page eligibility: review response status, redirects, canonical tags, robots and noindex signals, duplication, and whether the page is genuinely intended as a search landing page. A successful response alone does not make a URL suitable.
- Deduplicate and check scope: retain one copy of each normalized URL and ensure every entry belongs to the intended site scope.
- Use modification dates carefully: include a last-modified value only when it is reliably and verifiably accurate; do not invent dates to fill a field.
Screaming Frog says its default XML output includes internal HTML pages returning 200 and excludes redirects, errors, pages blocked by robots.txt, noindex pages, canonicalized URLs, paginated URLs, and PDFs. These are examples of that tool’s defaults, not a universal inclusion policy. Review the actual site’s canonicals, indexability, content, and business intent; the tutorial also documents excluding paths and removing unwanted rows.
Quick wins for a faster PC:
Scan for outdated or missing drivers - takes under a minuteDriver Scan →Clear out junk files and repair common Windows errorsFree Scan →Choose XML or a plain-text sitemap
XML is the versatile choice when you need sitemap extensions for images, video, news, or alternate-language pages. If you only need to provide page URLs, Google also accepts a plain-text sitemap with one full URL per line.
For XML, put each URL in a <loc> element inside a <url> entry, with entries enclosed by a <urlset> element. Escape XML-sensitive characters in tag values. Google ignores the <priority> and <changefreq> fields; its guidance says it uses <lastmod> when that value is consistently and verifiably accurate. Consult Google’s sitemap-building documentation for format details and supported extensions.
Validate and publish the file
- Validate the contents: confirm the file is well-formed in its chosen format, URLs are unique and within scope, and each entry follows your inclusion rules. Check status, canonical target, and indexability signals before deployment.
- Publish at a stable public URL: for example,
https://example.com/sitemap.xml. Use the site’s real hostname and actual published path. - Make the location discoverable: add a fully qualified directive such as
Sitemap: https://example.com/sitemap.xmlto the appropriate robots.txt, or submit the sitemap in Google Search Console. - Inspect processing: use Search Console’s Sitemaps report to check access and processing errors, then fix the underlying file or URL issues and resubmit if needed.
A robots.txt file’s rules apply only to the protocol, host, and port where that robots.txt is served. Its sitemap directive identifies a sitemap location; it does not grant permission to crawl URLs. Google’s robots.txt guidance explains the file’s scope.
Submitting a sitemap is a discovery hint, not a guarantee: Google says a sitemap can help it discover URLs but does not guarantee that the listed items will be crawled or indexed. See Google’s sitemap overview.
Recommended Free Tools
Best Value
Or skip the browser setup
If you need a rendered screenshot of a page while inspecting a site, ScreenshotNeo is a website screenshot API and MCP server for developers. It does not crawl a site or generate a sitemap; use the crawl workflow above for sitemap generation. For a screenshot, one GET request returns an image or PDF. This cURL example saves a WebP screenshot of Stripe:
curl -G "https://api.screenshotneo.com/v1/shot" -d access_key=YOUR_API_KEY --data-urlencode url=https://stripe.com -o shot.webpSee the ScreenshotNeo documentation for request options. Before capture, it accepts cookie or consent banners like a visitor and removes more than 60 known consent platforms, newsletter popups, and chat widgets; each step can be turned off. Bot checks, blank pages, and failed loads are never billed. Its MCP server lets AI agents use screenshot tools, and 1,000 screenshots per month are free with no card; paid plans start at $5 for 3,000 screenshots.
Sign up for 1,000 free screenshots a month with no card.
Frequently Asked Questions
Does submitting a sitemap make Google index every listed URL?
No. A sitemap helps Google discover URLs, but it does not guarantee crawling or indexing.
Outdated Drivers Are Slowing You Down
One free scan finds every outdated or missing driver and matches the right update for your exact hardware.Free scan · exact hardware matchPC Slower Than It Used to Be?
A free scan shows the junk files, broken settings and background clutter dragging Windows down - then fixes them in one click.Free scan · Windows 10 & 11Can I create a sitemap for a site I do not own?
This workflow is intended for sites you control or are authorized to crawl. The sitemap-specific guidance here does not establish permission to crawl third-party sites; respect their access controls and applicable policies.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
