Collect infinite-scroll results by repeatedly reading the rendered cards, adding only records with an unseen stable key to a persistent JavaScript Set, scrolling the correct container, and waiting for a meaningful sign of progress. Stop on the site’s end marker or after a bounded number of rounds without new unique records. The selectors, scroll target, and progress signal must match the page you are automating; there is no universal infinite-scroll selector.
Inspect the page before writing the loop
Identify the result-card selector, the fields to extract, and a stable identifier for each item. Also determine whether the document or an inner panel scrolls, whether the site exposes an end marker or loading indicator, and whether it removes older cards as new ones appear.
- Prefer an item ID: A site-provided result ID is usually a better deduplication key than visible text.
- Use URLs with care: If there is no stable ID, a normalized canonical URL can work. Tracking parameters may make the same destination look different, while a URL alone may not distinguish two legitimate results.
- Check the scroll owner: A results panel with its own scrolling will not respond to
window.scrollTo(). - Check for virtualization: If old cards disappear from the DOM as you scroll, save each batch immediately. A scrape of the final DOM will miss removed rows.
Puppeteer’s page.evaluate() runs code in the page context and awaits a returned promise. For reading every currently matching card, page.$$eval() passes the matches to a page-context function. Puppeteer recommends locators for selecting elements and interacting with them; locator actions wait for element presence and readiness. See the Page interactions guide, the Page.evaluate() reference, and the Page class reference.
Build a bounded collection loop
The example below is a runnable pattern once you replace the example selectors with the target page’s actual selectors. It assumes cards expose data-result-id, and that the document viewport scrolls. It records every unseen result as soon as it appears, waits for the card count to change, and stops at an end marker or after consecutive rounds produce no new records.
Quick wins for a faster PC:
Repair Windows errors before they cause bigger problemsFix Now →Scan for outdated or missing drivers - takes under a minuteDriver Scan →#1 Best Overall
- Plug-and-Play USB Camera Module: Experience ultimate convenience with our plug-and-play USB camera module. This 8MP camera is instantly recognized by Windows, Linux, Android, and macOS without any extra drivers. Just connect the USB and immediately start capturing crisp images, making it a perfect mini USB camera for rapid deployment in any project
- AI Resolution for Advanced Applications: Leverage multiple preset AI image resolutions to train and deploy your models seamlessly. This USB webcam and 3D printer camera eliminates the need for manual image cropping, delivering ready-to-process data straight from the sensor. It’s an ideal vision solution for developers and makers
- Autofocus & High-Definition Clarity: Equipped with a premium autofocus lens, this 4K mini camera automatically adjusts to maintain sharpness at various distances. Whether you’re using it as a lightburn camera for laser engraver or for detailed inspection, it delivers consistently clear and professional USB camera 4K quality video
- Robust & Reliable USB Security Camera: Built for durability and performance, this USB security camera offers steadfast monitoring with high-resolution imaging. Its versatile mounting and plug-and-play operation make it suitable for both home security setups and professional surveillance systems
- Upgraded Option with HDR: The enhanced model includes High Dynamic Range (HDR), an autofocus lens, and a rugged metal case. This upgraded USB camera module is especially suited for demanding applications like laser engraving with LightBurn or as a high-end 3D printer camera
const RESULT_SELECTOR = '.result-card';
const TITLE_SELECTOR = '.result-title';
const END_SELECTOR = '.end-of-results';
const MAX_NO_PROGRESS = 3;
const MAX_ROUNDS = 100;
const MAX_RESULTS = 10_000;
// Replace this with site-specific URL normalization if URLs are the only key.
function canonicalize(rawUrl) {
if (!rawUrl) return '';
try {
const url = new URL(rawUrl);
url.hash = '';
// Remove known tracking parameters only when they do not identify a result.
for (const key of [...url.searchParams.keys()]) {
if (/^(utm_.+|gclid|fbclid)$/i.test(key)) url.searchParams.delete(key);
}
return url.toString();
} catch {
return '';
}
}
const seen = new Set();
const results = [];
let noProgressRounds = 0;
for (let round = 0;
round < MAX_ROUNDS && noProgressRounds < MAX_NO_PROGRESS && results.length < MAX_RESULTS;
round++) {
const beforeCount = await page.$$eval(RESULT_SELECTOR, cards => cards.length);
const batch = await page.$$eval(RESULT_SELECTOR, cards => cards.map(card => ({
id: card.getAttribute('data-result-id') || '',
title: card.querySelector('.result-title')?.textContent?.trim() || '',
url: card.querySelector('a[href]')?.href || '',
})));
let added = 0;
for (const item of batch) {
const key = item.id ? `id:${item.id}` : `url:${canonicalize(item.url)}`;
if (key === 'url:' || seen.has(key)) continue;
seen.add(key);
results.push(item);
added++;
if (results.length >= MAX_RESULTS) break;
}
if (added === 0) noProgressRounds++;
else noProgressRounds = 0;
if (await page.$(END_SELECTOR)) break;
if (results.length >= MAX_RESULTS || round + 1 >= MAX_ROUNDS) break;
await page.evaluate(() => window.scrollTo(0, document.body.scrollHeight));
try {
await page.waitForFunction(
(selector, oldCount) => document.querySelectorAll(selector).length > oldCount,
{ timeout: 10_000 },
RESULT_SELECTOR,
beforeCount
);
} catch (error) {
// A timeout is not proof of completion: a request may be delayed, or the
// page may update a cursor/loading state without increasing the card count.
if (noProgressRounds >= MAX_NO_PROGRESS) break;
}
}
console.log(`Collected ${results.length} unique results`);
// Persist results here, especially if the page virtualizes and removes old cards.
The loop’s limits are safeguards, not universal settings. Choose values suitable for the page and the amount of data you intend to collect. A count increase is only a useful progress signal when the target page appends cards rather than replacing them or virtualizing a fixed set of rows.
Use the right key and preserve data
The Set lives in Node.js outside each DOM snapshot, so a card retained across multiple scrolls is only stored once. The example prefers an ID and falls back to a normalized URL. If an ID can be reused across different result types, combine it with the relevant type or scope. If URLs identify results only when a particular query parameter is present, preserve that parameter during normalization. Validate the chosen key against actual page records before relying on it.
Scroll the actual results region
For a nested scrolling panel, pass a selector for the panel and scroll that element instead of the window. For example, the page-context operation can be adapted to:
Rank #2
- ENGINEERED FOR AI & MOBILITY - Meet the Dell Pro 16 Plus, the AI-enhanced evolution of the Latitude 5550. Engineered for on-the-go productivity, it features a slim and lightweight design, delivers up to 11.9 hours of battery life, and supports ExpressCharge capability to keep you efficient. Boasting a durable aluminum chassis and having passed MIL-STD 810H tests, it offers robust reliability for professionals on the move, from the office to demanding field environments
- POWERFUL PERFORMANCE – The Dell Pro 16 Plus delivers power-efficient performance for demanding workloads with an AI PC powered by the AMD Ryzen AI 7 PRO 350 processor (up to 5.0GHz) and integrated Radeon 860M Graphics. Equipped with 32GB LPDDR5x RAM and 2TB M.2 NVMe PCIE SSD, enabling smooth multitasking and fast loading across a wide range of applications
- COPILOT+ PC AI POWERHOUSE - The dedicated NPU delivers 50 TOPS for local AI processing without relying on the cloud. It enables Recall (effortless retrieval of past actions and content), Cocreate (AI image tools), Windows Studio Effects (auto-framing/background blur for video calls), and Live Captions (real-time translation). It redefines productivity and creativity with seamless, offline AI acceleration
- IMMERSIVE DISPLAY - Features a 16-inch WUXGA (1920x1200) display with narrow borders, 300 nits brightness, and anti-glare coating to maximize screen real estate and reduce eye strain during extended use. Expand your workspace by connecting up to 3 external monitors via HDMI or Thunderbolt 4, with a max resolution of up to 4K@60Hz without docking station
- ADVANCED CONNECTIVITY -With Thunderbolt 4, USB-A, and HDMI 2.1, MicroSD card reader, Global Headset Jack and RJ45 Ethernet port, you can easily connect external displays, storage devices, and essential peripherals. Stay fast and reliable on the go with Wi-Fi 7 and Bluetooth 5.4, perfect for video calls, cloud work, and wireless devices without lag. The 1080p IR camera with temporal noise reduction ensures crisp video calls in any lighting and secure facial recognition login. Plus, the backlit keyboard enables precise typing in low-light environments
await page.$eval('.results-panel', panel => {
panel.scrollTop = panel.scrollHeight;
});Confirm the panel selector and that it owns the scroll. Some sites load more results when a particular sentinel enters view or when a “Load more” button is clicked. In that case, use the page’s actual trigger; a generic scroll may not activate it.
Do these 3 things before closing this tab:
1Fix the driver behind crashes, sound loss and screen glitches2Clear out junk files and repair common Windows errors3Scan for outdated or missing drivers - takes under a minuteWait on a condition, not just a guessed delay
page.waitForFunction() resolves when its predicate becomes truthy. Its options support animation-frame polling, DOM-mutation polling, or a numeric polling interval; the documented default timeout is 30 seconds and can be changed. See the FrameWaitForFunctionOptions reference. A result-count or cursor change is generally more informative than sleeping for a fixed duration, if that signal accurately reflects new content.
The sample uses a 10-second bounded wait as an example, not a guarantee that a particular site will respond within that time. If a site updates an existing card, changes a cursor, or clears a loading indicator without increasing the count, wait for that site-specific signal instead. A timeout should lead to a controlled retry or stopping decision, not an unbounded loop.
Rank #3
- Size: 5" x 4.6"
- Al weather vinyl sticker
- Phone sticker, laptop sticker, car sticker, water bottle sticker, and so many more applications!
- Peel & stick, simple application, reusable
- Made in the USA
Handle completion and no-progress cases
Use a clear site-specific end marker when available. Otherwise, stop after a chosen number of consecutive rounds that add no new unique records, while retaining an overall round or time limit. A round with no new records may mean the end has been reached, but it can also mean a delayed load, a wrong scroll target, or a weak progress condition.
- Reset the no-progress counter when at least one new unique key is added.
- Check an explicit end marker before deciding that a timeout means completion.
- Set a maximum number of rounds, results, or elapsed time so a broken page cannot run forever.
- Persist batches as you collect them when the DOM may discard earlier rows.
Troubleshoot common failures
Duplicates remain in the output
Check that the key is stable across renders and that the Set is outside the loop. Titles are often not unique. If using URLs, compare normalized versions and remove only parameters known to be tracking-only; stripping meaningful query parameters can merge distinct results.
Scrolling does not load anything
Inspect which element has scrollable overflow and target it. If the site loads on a sentinel or button rather than scroll position, wait for or interact with that element. Puppeteer locators are suited to interactive controls because they wait for elements to be ready before acting.
Rank #4
- For programmers and web developers who have a sense of gothic macabre about them. Perfect for coding meetups, gaming sessions, or casual outings. Do you live for code? Are you a programmer, IT professional or developer who is constantly coding?
- Web Developer Coding Skeleton In Front of Laptop Halloween. Perfect for dark mode developers, programmers, software engineers, anyone in tech with a dark side who lives at their computer. Great for Halloween or the rest of the year.
- Lightweight, Classic fit, Double-needle sleeve and bottom hem
The loop stops too soon
A timeout or unchanged card count alone does not prove that the page is finished. Allow the site’s loading indicator or request to settle, and wait on a reliable page-specific signal such as a cursor change or end marker. Increase the bounded wait only when the target’s behavior justifies it.
The loop never finishes
Ensure the end condition can actually match the page, and retain the maximum-round and no-progress limits. A page that continuously mutates unrelated DOM may make a broad mutation condition unhelpful; prefer a predicate tied to result count, cursor, or the relevant loading state.
Results are missing after long scrolling
The page may virtualize its list by removing cards outside the viewport. Extract and persist every batch before scrolling again rather than scraping once at the end. Also verify that the selector matches all result types and that the target page has finished rendering each batch before extraction.
PC Slower Than It Used to Be?
A free scan shows the junk files, broken settings and background clutter dragging Windows down - then fixes them in one click.Free scan · Windows 10 & 11Outdated Drivers Are Slowing You Down
One free scan finds every outdated or missing driver and matches the right update for your exact hardware.Free scan · exact hardware matchBest Value
- The Ultimate Developer Humor: Celebrate the fuel behind your best lines of code with this "I Turn Coffee Into Code" sticker. It is a must-have accessory for software engineers, web developers, data scientists, and computer science students.
- Premium Waterproof & Heat-Resistant: Crafted from high-quality, durable vinyl that is 100% waterproof and heat-resistant. Perfect for sticking on high-performance laptops, coffee tumblers, or office water bottles without worrying about peeling or fading.
- Sleek Professional Design: Featuring a bold black and white aesthetic with a clean coffee cup icon, this die-cut decal looks professional and stylish on MacBooks, PC cases, and office monitors.
- Easy Application, Zero Residue: Equipped with a strong adhesive that stays put through daily wear. If you upgrade your hardware, it peels off cleanly without leaving any sticky mess or residue behind on your expensive electronics.
- Perfect Tech Gift: Looking for a great gift for a programmer, IT professional, or coding student? This decal makes an excellent stocking stuffer, "new job" gift, or secret santa present for your tech-savvy coworkers.
Or skip the browser setup
ScreenshotNeo is a screenshot API and MCP server, not a replacement for extracting structured search-result records from a Puppeteer loop. If you need a screenshot of the page instead, one GET request can return an image or PDF. For example, this cURL request saves a WebP capture:
curl -G "https://api.screenshotneo.com/v1/shot" -d access_key=YOUR_API_KEY --data-urlencode url=https://stripe.com -o shot.webpSee the ScreenshotNeo API documentation for request options. Cookie banners are accepted and removed before capture, and known newsletter popups and chat widgets can be removed; those steps can be turned off. Bot checks, blank pages, failed loads, timeouts, and cache hits are not billed. Its MCP server lets AI agents use screenshot tools, and the free plan includes 1,000 screenshots a month with no card; paid plans start at $5 for 3,000 screenshots.
Sign up free for ScreenshotNeo to get 1,000 screenshots a month with no card.
FAQ
Can I scrape all infinite-scroll results from the final DOM?
Not reliably. A virtualized page may remove earlier cards as you scroll, so capture each batch while it is present.
The Tool Desk
Outbyte Driver Updater FREEScan for outdated or missing drivers - takes under a minuteDriver Scan →Outbyte PC Repair FREEClear out junk files and repair common Windows errorsFree Scan →Does Puppeteer provide a universal selector or end condition for search results?
No. The target site determines its card selectors, scroll behavior, stable key, progress signal, and completion marker.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
