PC Slower Than It Used to Be?
A free scan shows the junk files, broken settings and background clutter dragging Windows down - then fixes them in one click.Free scan · Windows 10 & 11Outdated Drivers Are Slowing You Down
One free scan finds every outdated or missing driver and matches the right update for your exact hardware.Free scan · exact hardware matchiTechGuides is reader-supported. When you buy through links on our site, we may earn an affiliate commission. As an Amazon Associate I earn from qualifying purchases. Learn more
Plain cURL cannot download an entire website by following its links. The curl project FAQ says its command-line tool has no built-in recursive operation. For a conventional, authorized mirror of pages linked in HTML or CSS, use GNU Wget’s recursive options instead. If you must use cURL, it can fetch individual URLs, but you need to supply those URLs yourself or build a crawler around libcurl.
Can cURL download an entire website?
No—not by itself. cURL transfers data from URLs you give it; it does not discover links on a page and recursively fetch them. The curl project FAQ explicitly distinguishes cURL from Wget and other tools that perform recursive retrieval. A shell loop can fetch a known list of URLs, and a program using libcurl can be made into a crawler, but neither is a built-in recursive cURL command.
That distinction matters because downloading one HTML page is not the same as making an offline website copy. A page may link to other pages, stylesheets, images, scripts, and files. A recursive mirror needs to discover and retrieve those resources, keep its crawl within the intended scope, and, if you want to browse locally, adjust links to refer to the downloaded files.
Use cURL for individual URLs
For a single page, cURL can save the response body to a file:
#1 Best Overall
- HTML CSS Design and Build Web Sites
- Comes with secure packaging
- It can be a gift option
curl -L "https://example.com/docs/" -o page.html-L follows HTTP redirects; -o writes the response body to the named file. This does not find or download links in the page, and page.html may not display as it does online if it depends on assets you have not fetched.
When a custom crawler makes sense
If the task requires rules beyond a conventional mirror—such as a precise URL allowlist, custom handling of a site’s application routes, or special storage and retry behavior—you can write a crawler using libcurl. The link-discovery, scope, deduplication, and file-mapping logic would be your program’s responsibility. That is a larger project than invoking cURL, and its behavior depends on how you implement it.
Use Wget for a conventional linked-site mirror
For a permitted site whose pages are discoverable from the starting URL, GNU Wget is the practical command-line choice. Its recursive retrieval can parse HTML and CSS links; its mirror option bundles recursive retrieval and infinite recursion depth with other mirror settings. Try this starting command on a section you are allowed to archive:
wget --mirror --convert-links --adjust-extension --page-requisites --no-parent --wait=1 https://example.com/docs/Replace the example URL with the intended starting page or directory. Run the command in a directory where you have room for the files. This is a starting configuration, not a promise of a complete clone: what Wget finds depends on what it can discover through the content it retrieves.
What each option does
| Option | Purpose | Practical implication |
|---|---|---|
--mirror |
Enables recursive retrieval with infinite recursion depth and turns on timestamping. | A crawl can continue through all discoverable levels, so inspect scope before using it on a large site. |
--page-requisites |
Fetches resources needed to display a page, such as referenced stylesheets and inline images. | It retrieves display assets; it is not the option that recursively visits every linked page. |
--no-parent |
Prevents retrieval above the selected hierarchy. | Choose the starting URL carefully. Starting at a section can help limit the mirror to that area. |
--convert-links |
Converts downloaded links for local use. | Useful when the goal is to browse the saved copy offline rather than merely retain downloaded files. |
--adjust-extension |
Adjusts local file extensions as needed for compatibility. | Behavior can vary with the installed Wget release; verify it for your version and workflow. |
--wait=1 |
Waits one second between accesses. | A delay can reduce the chance of overloading a public server, though it does not by itself establish permission to crawl. |
Set the scope before you start
Choose the correct starting URL
The start URL determines what Wget can discover, while --no-parent helps avoid moving above that URL’s hierarchy. If you want a documentation section rather than an entire domain, start at that section’s URL and review the saved paths before expanding the crawl. A URL’s apparent directory structure does not guarantee that every page belongs to the same content boundary, so check the results.
Begin small, then expand
Wget’s documented default recursive depth is five layers; --mirror sets infinite depth. To keep an initial test bounded, you can use a finite depth instead of --mirror, for example:
Rank #2
wget --recursive --level=2 --convert-links --page-requisites --no-parent --wait=1 https://example.com/docs/Inspect which URLs and files were saved, whether internal links work locally, and how much disk space the result uses. Only then decide whether to broaden the crawl. A shallow test is also a way to detect that the start URL reaches unrelated parts of a site.
Free tools Windows power users keep installed
One-click scans. No signup required.
Respect access rules and server capacity
GNU Wget’s overview says it respects the Robot Exclusion Standard, but that does not decide whether a particular crawl is permitted. Check the site’s applicable access rules and obtain authorization where required. Recursive retrieval can place load on the remote server and consume local disk, bandwidth, memory, and CPU. Keep the scope appropriate, use a delay for public servers, and watch the destination storage while the command runs.
What a Wget mirror can miss
Wget’s documented traversal model follows discoverable references in HTML, XHTML, and CSS. A link-based crawl cannot guarantee that it will reproduce every website, particularly where content depends on behavior outside that model.
- Client-side JavaScript: Pages or links created only after scripts run may not be discovered as they would be by an interactive browser.
- Authenticated flows: A crawl of public links does not automatically reproduce a logged-in session or every access-controlled route. Do not assume a public mirror command will archive a password-protected site.
- API-driven content: Data loaded from application APIs may not appear as ordinary links in the retrieved HTML or CSS.
- Unlinked pages: A page that cannot be reached from the starting URL through links Wget parses will not be found merely because it exists on the domain.
For these cases, first determine what you are authorized to access and what “entire website” means for your purpose. A link-based offline mirror, a record of selected pages, a database export, and a fully functioning copy of a dynamic web application are different outcomes; Wget’s recursive options target the first kind of task.
cURL, Wget, or libcurl: which approach fits?
| Approach | Discovers and follows page links? | Best fit | Main trade-off |
|---|---|---|---|
| Plain cURL | No built-in recursive operation. | Fetching one URL or a known list of URLs. | You must provide each URL; it does not make an offline mirror by itself. |
| GNU Wget recursion | Yes, for discoverable HTML and CSS references. | A conventional linked-site mirror with documented scope controls. | Dynamic, authenticated, API-driven, or unreachable content may be missed. |
| Custom program using libcurl | Possible if you implement it. | Workflows needing custom discovery or storage rules. | You must design and maintain the crawler behavior yourself. |
For a typical static or mostly link-based section, start with Wget and a narrow scope. Choose cURL when you already know the URLs and need to retrieve them individually. Choose a custom libcurl program only when its added control justifies implementing the crawling logic.
Troubleshooting common problems
The command downloaded only one page
If you ran cURL, that is expected: it does not recursively follow page links. Use Wget’s recursive options for a crawl, or supply a list of URLs to cURL yourself.
The offline page is missing styles or images
Confirm you used --page-requisites. That option fetches resources needed to display pages, while --convert-links helps make downloaded links usable locally. Neither option guarantees that JavaScript-only or API-loaded content will be reproduced.
The crawl includes too much—or too little
Review the starting URL and --no-parent scope. If the mirror is too broad, test a narrower starting section or use a finite depth such as --level=2. If it is too shallow, remember that ordinary recursive retrieval defaults to five layers, whereas --mirror uses infinite depth. Confirm that the missing pages are actually linked in material Wget parses.
Some pages are blank or incomplete offline
Check whether the live page relies on client-side JavaScript, a logged-in session, or API responses. A link-based mirror may save the HTML and referenced files without reproducing the application behavior that populates the page. Wget’s traversal model does not establish that such content has been captured.
The download is slow or uses too much storage
A recursive mirror can retrieve many pages and assets. Narrow the starting scope, use a finite depth for testing, and retain a delay such as --wait=1 when accessing a public server. Monitor disk usage and stop if the crawl is exceeding the authorized or practical scope.
Local links do not work as expected
Use --convert-links when you intend to browse the downloaded copy, and inspect representative pages and links after retrieval. Also check the installed Wget version’s behavior for --adjust-extension; the Wget manual cited for these options is version 1.25.0, and distributions may package different releases.
Or skip the browser setup
ScreenshotNeo is a screenshot API, not a recursive website downloader: it captures a URL as an image or PDF rather than saving a browsable copy of every linked page. If your actual goal is a clean visual capture of a page, one GET request can return a screenshot. See the ScreenshotNeo API documentation for options and response details.
Rank #4
curl -G "https://api.screenshotneo.com/v1/shot" -d access_key=YOUR_API_KEY --data-urlencode url=https://stripe.com -o shot.webpIn this example, replace the target URL as needed and use your API key. The same endpoint can be called from Python or Node.js:
Do these 3 things before closing this tab:
1Clear out junk files and repair common Windows errors2Fix the driver behind crashes, sound loss and screen glitches3Repair Windows errors before they cause bigger problemsimport requests
r = requests.get("https://api.screenshotneo.com/v1/shot", params={"access_key": "YOUR_API_KEY", "url": "https://stripe.com"}, timeout=90)
open("shot.webp", "wb").write(r.content)const q = new URLSearchParams({ access_key: 'YOUR_API_KEY', url: 'https://stripe.com' });
const res = await fetch(`https://api.screenshotneo.com/v1/shot?${q}`);- Cookie and consent banners are accepted like a visitor, and more than 60 known consent platforms, newsletter popups, and chat widgets are removed before capture; each of those steps can be turned off.
- Bot checks or CAPTCHAs, blank pages, timeouts, failed loads, and cache hits cost nothing; response headers report the page verdict and billing status.
- An MCP server provides
take_screenshot,get_page_info, andcapture_pdftools for AI agents and MCP clients. - The free plan includes 1,000 screenshots per month with no card; paid plans start at $5 for 3,000 shots.
Sign up for ScreenshotNeo’s free plan to try it with 1,000 screenshots a month and no card.
Version note
The Wget option descriptions and depth behavior above refer to the GNU Wget 1.25.0 manual. Check the manual and help output for the version installed on your operating system before relying on version-sensitive behavior, particularly extension adjustment. The cURL distinction is stated in the curl project FAQ.
Frequently Asked Questions
Will a Wget mirror keep the original website’s domain and redirects?
It saves files for local browsing and can convert downloaded links; it is not a live copy hosted under the original domain. The example command does not recreate the site’s hosting, routing, or server-side redirect behavior.
Can I use a recursive download to make a backup of a site I do not own?
Not automatically. GNU Wget’s support for the Robot Exclusion Standard does not determine whether your intended use is permitted. Check applicable site rules and obtain authorization where required.
What’s actually slowing this PC down?
Pick the symptom - the matching free tool is one click away.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
