Recommended Free Tools
iTechGuides is reader-supported. When you buy through links on our site, we may earn an affiliate commission. As an Amazon Associate I earn from qualifying purchases. Learn more
There is no reliable “every page” button. To create a defensible screenshot archive, define an explicit scope, build a URL queue from the XML sitemap and internal links, normalize and deduplicate that queue, crawl politely with a real browser, save full-page images, and publish a manifest showing what was captured, redirected, skipped, or failed. A sitemap or successful browser run is not proof that undiscovered URLs do not exist.
1. Define what “every page” means
Write the scope before writing code. Record the starting origin (for example, https://example.com), whether subdomains count, URL paths to include or exclude, whether query-string variants are distinct pages, and whether authenticated areas are allowed. Never bypass a login, paywall, robots restriction, rate limit, or another access control.
- Host: decide whether
www.example.com,docs.example.com, and other subdomains are separate. - Protocols and redirects: retain both the requested URL and the final URL after navigation.
- URL rules: define policies for fragments, trailing slashes, tracking parameters, print views, calendars, search results, and other potentially infinite patterns.
- Authentication: use only credentials and sessions you are authorized to use; otherwise keep private routes out of scope.
- Capture goal: choose a viewport image or a full-page image of the complete scrollable document.
State these choices in the final report. “Every page” should mean every URL discovered and permitted within this written scope, not an unprovable claim about the whole site.
Outdated Drivers Are Slowing You Down
One free scan finds every outdated or missing driver and matches the right update for your exact hardware.Free scan · exact hardware matchWindows Errors? Fix Them Before They Spread
Repair common Windows errors and clear accumulated junk for a smoother, more stable PC - no reinstall needed.Free scan · no reinstall2. Build a URL inventory
Use the sitemap as a declared inventory
Fetch /sitemap.xml and any sitemap index it references. Treat each listed location as an input to your own queue. A sitemap helps search engines discover URLs, but it does not guarantee that every listed item will be crawled or indexed; the same limitation applies to your capture job.
#1 Best Overall
- MADE FOR THE MAKERS: Create; Explore; Store; The T7 Portable SSD delivers fast speeds and durable features to back up any endeavor; Build your video editing empire, file your photographs or back up your blogs all in an instant
- SHARE IDEAS IN A FLASH: Don’t waste a second waiting and spend more time doing; The T7 is embedded with PCIe NVMe technology that brings fast read and write speeds up to 1,050/1,000 MB/s¹, making it almost twice as fast as the T5
- ALWAYS MAKE THE SAVE: Compact design with massive capacity; With capacities up to 4TB, save exactly what you need to your drive – from large working files to game data and everything in between
- ADAPTS TO EVERY NEED: Whether using a PC or mobile phone, count on the T7 for extensive compatibility²; It’s a true team player when it comes to heavy-duty application usage or file-saving
- HI RESOLUTION VIDEO RECORDING: Record Ultra High Resolution (4K 60fs) videos directly onto the T7 Portable SSD with your favorite camera or mobile devices; Supports iPhone 15 Pro Res 4K at 60fps video and more³
Discover links from pages
Starting with the home page, follow same-scope links in the rendered DOM. This finds routes missing from an outdated sitemap and can expose links inserted by JavaScript. It still cannot reveal pages that are not linked or listed, so combine it with the sitemap whenever possible.
Normalize and deduplicate
Resolve relative links against the page that contained them, remove fragments unless your application treats them as separate server-rendered documents, and apply your query-parameter policy consistently. Preserve meaningful parameters. Store a canonical queue key while retaining the original requested URL for auditing.
3. Check robots.txt and crawl gently
Read the site’s robots.txt and honor applicable restrictions for the crawler you operate. Robots.txt communicates crawler-access preferences and can help manage traffic; it is not a privacy or security boundary and does not make a URL confidential. Documentation about Google’s crawlers describes Google’s process, not a universal legal rule for every private capture, so obtain permission where required.
Quick wins for a faster PC:
Clear out junk files and repair common Windows errorsFree Scan →Fix the driver behind crashes, sound loss and screen glitchesFind Drivers →Repair Windows errors before they cause bigger problemsFix Now →Start with low concurrency, add a delay between requests, and reduce parallelism when response times rise or the server returns errors. Bounded retries are appropriate for transient failures; endless retries can become a denial-of-service pattern.
Rank #2
- Get NVMe solid state performance with up to 1050MB/s read and 1000MB/s write speeds in a portable, high-capacity drive(1) (Based on internal testing; performance may be lower depending on host device & other factors. 1MB=1,000,000 bytes.)
- Up to 3-meter drop protection and IP65 water and dust resistance mean this tough drive can take a beating(3) (Previously rated for 2-meter drop protection and IP55 rating. Now qualified for the higher, stated specs.)
- Use the handy carabiner loop to secure it to your belt loop or backpack for extra peace of mind.
- Help keep private content private with the included password protection featuring 256‐bit AES hardware encryption.(3)
- Easily manage files and automatically free up space with the SanDisk Memory Zone app.(5). Non-Operating Temperature -20°C to 85°C
4. A complete Playwright crawler and screenshot script
The following Node.js example combines sitemap and link discovery, captures a full scrollable page, records redirects and errors, and writes a JSON manifest. Install Playwright with npm install playwright, then run the script with a permitted origin.
const { chromium } = require('playwright');
const fs = require('fs/promises');
const crypto = require('crypto');
const ORIGIN = new URL(process.argv[2] || 'https://example.com');
const MAX_PAGES = Number(process.env.MAX_PAGES || 1000);
const DELAY_MS = Number(process.env.DELAY_MS || 500);
const OUT = 'screenshots';
function inScope(url) {
return url.origin === ORIGIN.origin && !url.username && !url.password;
}
function normalize(raw, base) {
const u = new URL(raw, base);
u.hash = '';
// Keep meaningful query parameters; remove only parameters your policy defines as tracking.
for (const key of [...u.searchParams.keys()]) {
if (/^(utm_|gclid$|fbclid$)/i.test(key)) u.searchParams.delete(key);
}
return u.href;
}
function fileName(url) {
return crypto.createHash('sha256').update(url).digest('hex').slice(0, 20) + '.png';
}
const sleep = ms => new Promise(r => setTimeout(r, ms));
(async () => {
await fs.mkdir(OUT, { recursive: true });
const browser = await chromium.launch();
const page = await browser.newPage({ viewport: { width: 1440, height: 900 }, deviceScaleFactor: 1 });
const queue = new Set([ORIGIN.href]);
const queued = new Set();
const manifest = [];
// Sitemap discovery. A sitemap index is followed recursively.
async function readSitemap(url, seen = new Set()) {
if (seen.has(url)) return;
seen.add(url);
try {
const res = await page.request.get(url, { timeout: 30000 });
if (!res.ok()) return;
const xml = await res.text();
for (const match of xml.matchAll(/<loc>\s*([^<]+?)\s*<\/loc>/gi)) {
const loc = normalize(match[1], url);
if (!inScope(new URL(loc))) continue;
if (xml.includes('<sitemapindex')) await readSitemap(loc, seen);
else queue.add(loc);
}
} catch (_) { /* the home-page crawl remains usable */ }
}
await readSitemap(new URL('/sitemap.xml', ORIGIN).href);
while (queue.size && queued.size < MAX_PAGES) {
const requested = queue.values().next().value;
queue.delete(requested);
if (queued.has(requested)) continue;
queued.add(requested);
const record = { requestedUrl: requested, capturedAt: new Date().toISOString(), status: null, finalUrl: null, file: null, error: null };
try {
const response = await page.goto(requested, { waitUntil: 'networkidle', timeout: 60000 });
record.status = response ? response.status() : null;
record.finalUrl = page.url();
const final = new URL(record.finalUrl);
if (!inScope(final)) throw new Error('redirected outside the configured origin');
record.file = fileName(requested);
await page.screenshot({ path: `${OUT}/${record.file}`, fullPage: true });
for (const href of await page.locator('a[href]').evaluateAll(as => as.map(a => a.href))) {
try {
const normalized = normalize(href, record.finalUrl);
if (inScope(new URL(normalized)) && !queued.has(normalized)) queue.add(normalized);
} catch (_) {}
}
} catch (error) {
record.error = String(error.message || error);
}
manifest.push(record);
await sleep(DELAY_MS);
}
await fs.writeFile('manifest.json', JSON.stringify({ origin: ORIGIN.href, attempted: manifest.length, remainingQueue: queue.size, pages: manifest }, null, 2));
await browser.close();
})();page.screenshot({ path, fullPage: true }) captures the full scrollable document rather than only the current viewport. It does not discover additional URLs; link discovery in the example is a separate step. For very long pages, the resulting image can be extremely tall, so consider a viewport capture or a PDF for review workflows.
5. Make the run auditable
Keep the manifest beside the images. At minimum, store:
Do these 3 things before closing this tab:
1Scan for outdated or missing drivers - takes under a minute2Clear out junk files and repair common Windows errors3Fix the driver behind crashes, sound loss and screen glitches- requested URL and final URL;
- capture timestamp and HTTP status;
- stable filename;
- error text, if any;
- the scope, exclusions, viewport, browser version, and script revision.
Use a hash-based filename or another safe identifier instead of raw paths, which can contain characters invalid on some filesystems. Retry a small, fixed number of transient navigation failures, then leave the record marked failed. Never silently drop a URL.
Rank #3
- Capacity Display Variance: 250GB external ssd often appears as around 232GB on Windows. MacOS can show full 250 GB capacity. This is binary calculation difference and doesn’t affect SSD hard drive actual physical storage
- 1050 MB/s Speed: Instantly access to your files with blazing-fast 10Gbps external SSD read up to 1050MB/s and write up to 1000MB/s. LED Light indicates USB SSD instant activity
- Data Security: Solid state drives S.M.A.R.T. health diagnostics and adaptive TRIM optimizing data block management ensures consistent write speeds and extends the longevity of the portable SSD
- USB-C & USB-A Cable: Both cables featuring rapid USB 3.2 Gen2, this USB SSD effortlessly bridges devices, enabling seamless cross-platform file transfers and backup between computers, smartphones, tablets and iPhone
- Always Fast: No slowdowns for large file transfers. With SLC caching (25% of current available capacity allocated as high-speed cache), this external SSD delivers steady 10Gbps for transfers within the cache capacity
6. Choosing discovery and capture strategies
| Strategy | Strength | Weakness | Best use |
|---|---|---|---|
| Link-only crawl | Finds links rendered in pages | Misses unlinked or orphaned routes | Small, navigation-driven sites |
| Sitemap-only list | Fast access to the site’s declared inventory | Can be stale, incomplete, or contain URLs that fail | Sites with a carefully maintained sitemap |
| Combined sitemap plus links | Explains more omissions and broadens discovery | Requires deduplication and clear scope rules | Most defensible general workflow |
For captures, choose a viewport screenshot when you need a consistent visual state at a known screen size. Choose full-page mode when reviewers need all scrollable content in one image, accepting larger dimensions and more difficult visual comparison.
7. Common failures and fixes
Timeouts or blank pages
Increase the navigation timeout only when the site is genuinely slow; first check DNS, TLS, server health, and whether the page waits forever on third-party requests. Record the failure and retry later rather than looping indefinitely.
Bot checks or CAPTCHAs
Do not attempt to evade a challenge. Obtain permission, use an authorized test environment, or exclude the route and document the exclusion.
Redirect leaves the scope
Compare the final origin with the configured origin. Keep the requested and final URLs in the manifest and mark the route out of scope if the redirect is external.
Rank #4
- Easily store and access 2TB to content on the go with the Seagate Portable Drive, a USB external hard drive
- Designed to work with Windows or Mac computers, this external hard drive makes backup a snap just drag and drop
- To get set up, connect the portable hard drive to a computer for automatic recognition no software required
- This USB drive provides plug and play simplicity with the included 18 inch USB 3.0 cable
- The available storage capacity may vary.
Duplicate screenshots
Apply one normalization policy before enqueueing. Decide explicitly whether trailing slashes and meaningful query parameters identify different resources.
JavaScript links are missing
Wait for the page’s required state or a selector before extracting links. If the application loads routes only after interaction, add an authorized, deterministic click step and record it.
Memory or storage pressure
Limit the queue, stream or periodically archive the manifest, close unused browser contexts, and avoid retaining image buffers in memory. Full-page images consume substantially more storage than viewport images.
8. Performance, reliability, and coverage reporting
Concurrency is a trade-off: more workers increase throughput but also server load, memory use, and the chance of rate limiting. Begin with one worker, measure response behavior, then increase cautiously. Use a bounded retry policy with backoff for connection resets and 5xx responses, but do not retry deterministic 4xx errors forever.
Best Value
- NEARLY 2X FASTER THAN OUR PREVIOUS GENERATION(8) – move 1,000 high-res photos in under 60 seconds(6) with up to 2000MB/s transfer speeds(2).
- IP65 RATING AND UP TO 3M DROP PROTECTION(3) – protects against spills and drops.
- POCKET-SIZED – fits easily in pockets and small bags.
- SPACE TO OWN YOUR AI CONTENT – speed and capacity to download your high-res clips and photo edits.
- 256-BIT AES ENCRYPTION(4) – helps keep private files secure with password protection.
At the end, report the number attempted, captured, failed, skipped by rule, redirected, and still queued. Include discovery sources (sitemap files and link traversal), crawl date, scope, exclusions, and the manifest location. A complete sitemap export or a successful browser run is evidence of what you processed, not proof that no undiscovered page exists.
Or skip the browser setup
ScreenshotNeo provides a website screenshot API and MCP server. One request returns a PNG, JPEG, WebP, or PDF; you still supply the URL inventory and can submit URLs individually or with its bulk capture option (up to 100 URLs per call).
It removes cookie/consent banners, newsletter popups, and chat widgets before capture. Bot checks/CAPTCHAs, blank pages, timeouts, failed loads, and cache hits are not billed, and response headers identify the page verdict and whether it was billed. An MCP server exposes take_screenshot, get_page_info, and capture_pdf to Claude, Cursor, and other MCP clients.
Free tools Windows power users keep installed
One-click scans. No signup required.
See the ScreenshotNeo API documentation for all options, including full-page capture, CSS selectors, device presets, retina scale, waits, custom headers and cookies, blocking rules, geolocation, signed links, asynchronous jobs, webhooks, caching, and usage reporting.
cURL
curl -G "https://api.screenshotneo.com/v1/shot" -d access_key=YOUR_API_KEY --data-urlencode url=https://stripe.com -o shot.webpPython
import requests
r = requests.get("https://api.screenshotneo.com/v1/shot", params={"access_key": "YOUR_API_KEY", "url": "https://stripe.com"}, timeout=90)
open("shot.webp", "wb").write(r.content)Node.js
const q = new URLSearchParams({ access_key: 'YOUR_API_KEY', url: 'https://stripe.com' });
const res = await fetch(`https://api.screenshotneo.com/v1/shot?${q}`);The Free plan includes 1,000 screenshots per month with no card. Paid plans start at $5 for 3,000 shots; every feature is available on every plan. Create a free ScreenshotNeo account to get started.
Frequently Asked Questions
Can a crawl prove that a site has no other pages?
No. It can document every URL found through the stated sitemap, link, and scope rules, while clearly listing exclusions and failures.
Should URL fragments be captured separately?
Usually no, because fragments are handled in the browser and are not sent to the server; make a different policy only when your application renders distinct fragment states that matter to the archive.
Is a PDF better than a full-page image?
Use a PDF when pagination, selectable text, or print-oriented review matters. Use a full-page image for a single visual record of the scrollable document.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
