Free tools Windows power users keep installed
One-click scans. No signup required.
iTechGuides is reader-supported. When you buy through links on our site, we may earn an affiliate commission. As an Amazon Associate I earn from qualifying purchases. Learn more
A broad robots.txt rule can stop Google from crawling unwanted URL patterns, but it does not remove URLs from search results or guarantee better indexing. In a September 2026 account on DEV Community, Toolore’s author said an inherited domain produced 957,000 known URLs for a new 66-page site. The author blocked query-string URLs because the new site did not use them; that was a site-specific fix, not a safe rule to copy without checking what your own URLs do.
How 66 pages turned into 957,000 reported URLs
Toolore’s author described launching a static Next.js site on a domain that had previously been an e-commerce store. Six weeks later, Google Search Console showed 957,000 known URLs, while the sitemap listed 66 discovered pages. The author’s examples included old query-string URLs that now served the new homepage with an HTTP 200 response. These are figures from the author’s Search Console view, not an independently audited count or a general benchmark. Toolore’s account
The author reported 621,728 URLs classified as “Crawled – currently not indexed,” 335,238 as “Soft 404,” 358 as “Not found (404),” 65 as “Discovered – currently not indexed,” and one indexed page. The proposed diagnosis was that many query-string URLs displayed the homepage despite representing no real page, while some old product paths correctly returned 404 responses. Those are different cases: the author left the genuine missing product URLs as 404s rather than blocking them as query URLs. Toolore’s account
Windows Errors? Fix Them Before They Spread
Repair common Windows errors and clear accumulated junk for a smoother, more stable PC - no reinstall needed.Free scan · no reinstallCrashes, No Sound, or Screen Glitches?
Random freezes, missing sound and display glitches usually trace back to one bad driver. Find and replace yours safely.Free scan · under a minuteWhy a homepage response can become a soft 404
Google defines a soft 404 as “a URL that returns a page telling the user that the page does not exist and also a 200 (success) status code.” A URL that does not have meaningful content but returns the homepage with a successful status can fit that pattern. Google’s guidance is to return an appropriate HTTP status for missing content, rather than making an unrelated page appear to satisfy the request. Google Search Central: Troubleshoot Google Search Crawling Errors
#1 Best Overall
A sitemap reporting success does not explain why URLs appear in the Pages report or why Google classifies them as it does. Inspect examples from Search Console and group them by path and parameter pattern. Check what each URL returns and whether it represents a real page, a removed item, a redirect, or an accidental duplicate. The important distinction is the URL’s intended behavior—not just its appearance in a report.
What the robots.txt rule did in this case
The author’s example was:
User-Agent: *
Allow: /
Allow: /icon.svg
Disallow: /*?
Sitemap: https://toolore.com/sitemap.xmlThe Disallow: /*? line tells compliant crawlers not to crawl URLs whose path includes a question mark. Toolore’s author said the new site had no pages using query parameters. The separate /icon.svg allow rule was included because the favicon URL used a cache-busting query string. This configuration reflects the author’s site; it is not a universal template. Toolore’s account
Check legitimate uses before blocking parameters
Query strings can power real features, including search results, sorting, and filters. Google’s URL-structure guidance discusses managing problematic dynamic URL spaces, but a site should first identify which parameter patterns serve useful content or assets. A blanket block could prevent Google from crawling pages you want indexed. Google Search Central: URL Structure Best Practices for Google Search
- Review representative URLs in Search Console and group them by pattern.
- Test whether each pattern serves a distinct, useful page or only duplicates another response.
- Check query-string use in site search, filters, sorting, tracking, and assets such as favicons.
- Block only patterns you have established should not be crawled; preserve any required exceptions.
Choose the fix for the outcome you need
Robots rules, noindex directives, and HTTP status codes solve different problems. Choose based on whether you need to stop crawling, keep accessible content out of Google Search, or report that content is gone. Google Search Central: Robots.txt Introduction and Guide
Rank #3
| Situation | Appropriate response | Key limitation |
|---|---|---|
| You want to stop crawling a problematic URL pattern. | Use a carefully scoped robots.txt rule after checking for legitimate pages or assets that use the pattern. | Blocking crawling does not reliably remove a URL from search results; Google may index a blocked URL if it finds links to it. |
| The page remains accessible but should not appear in Google Search. | Use a noindex directive and allow Google to crawl the page so it can read that directive. |
If robots.txt blocks the URL, Google cannot read the page’s noindex instruction. |
| The content is gone and has no comparable replacement. | Return HTTP 404 or 410. | Do not redirect an unrelated, missing URL to the homepage as a substitute for a real response. |
| The old page has a clear, relevant replacement. | Use a permanent redirect to that replacement. | A redirect is appropriate only when the destination genuinely replaces the old content. |
Google’s documentation explains that blocked URLs can remain eligible for indexing when other pages link to them, and that a noindex directive must be crawlable for Google to see it. Google Search Central: Block Search Indexing with noindex
What robots.txt does not promise about crawl budget
Blocking unwanted URLs may be useful for crawl management, but it does not automatically redirect saved crawling to important pages. Google states: “Blocking or hiding already crawled pages from recrawls won’t shift your crawl budget to another part of your site unless Google is already hitting your site’s serving limits.” Google also cautions that blocked URLs can remain in its crawl queue longer and may be recrawled if the block is later removed. Google for Developers: Crawl Budget Management
Independent reader supportYour contribution helps us test, update, and keep practical guides available for everyone.What changed after the rule—and what remains unknown
Toolore’s author cautioned that it was too early to call the change a win because Search Console data lagged by several days. In the snapshot described in the post, “Discovered – currently not indexed” had moved from 65 to 55, while the indexed count remained one. That self-reported movement does not establish that the robots rule caused the change, demonstrate a ranking recovery, or show how long a similar issue takes to resolve elsewhere. Toolore’s account
The Tool Desk
Outbyte PC Repair FREEClear out junk files and repair common Windows errorsFree Scan →Outbyte Driver Updater FREEScan for outdated or missing drivers - takes under a minuteDriver Scan →The author also changed sitemap lastmod values from the site’s build time to each page’s content update date. Google recommends using lastmod to indicate when a page changed, but a sitemap is a hint, not a guarantee of immediate crawling or indexing. The post mentions submitting URLs through IndexNow; the cited Google guidance does not verify a faster-discovery effect from that action. Google Search Central: Troubleshoot Google Search Crawling Errors
When to use Google’s Removals tool
Google’s Removals tool is temporary: a successful request lasts about six months. It can be useful when information needs to disappear from results quickly, but it does not replace an enduring fix. For permanent handling, use the appropriate status code, redirect, or crawlable noindex directive for the URL’s actual situation. Google Search Central: Remove Your Site Info from Google
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
