If Puppeteer scrolls without loading more results, waits forever, or keeps seeing the same content, first identify the element that actually scrolls. Then wait for a page-specific sign of progress—such as a higher item count or a loader disappearing—instead of relying on a fixed delay. Keep the loop bounded and stop when the site signals that results are finished.
Why infinite-scroll scripts stop or stall
Infinite scrolling is site-specific: a page may load more content when the document reaches a threshold, when a nested panel scrolls, or when a loading trigger becomes visible. Scrolling the wrong element will not activate that behavior. Even when the scroll is correct, the next request may take longer than expected, fail, or return no new items.
Puppeteer’s page.evaluate runs a function in the browser page context and waits for a returned Promise to resolve. Puppeteer’s evaluate API documentation reports version 25.12.0; check the API docs for the version installed in your project because signatures can change.
Build a progress-based scrolling loop
- Find the scrolling element. Determine whether the page uses the document or a nested container. In DevTools, inspect likely panels and their computed
overflowand scroll dimensions; a container withscrollHeight > clientHeightmay be scrollable. - Choose a progress signal. Examples include an increased count of rendered cards, a new item ID, a changed next-page token exposed in the UI, or a loader disappearing. For virtualized lists, use stable item IDs or URLs when possible; rendered row count may stay constant as rows are recycled.
- Scroll the correct target. Use a controlled increment if the site loads near a threshold, or scroll to the current end if that is how its trigger works.
- Wait for the expected change. Use
page.waitForFunctionfor a page-context predicate, or a locator wait for an element state that matches the expected outcome. The Page API documentation describes the wait APIs; Puppeteer’s interactions guide explains locator behavior. - Bound the work. Set a maximum round count or elapsed-time budget, track consecutive no-progress rounds, and define a site-specific end-of-results condition.
The official API docs provide the evaluation and wait primitives; they do not prescribe one universal infinite-scroll loop. Adapt the selector, trigger, and stop condition to the page you are automating.
What’s actually slowing this PC down?
Pick the symptom - the matching free tool is one click away.
#1 Best Overall
Illustrative JavaScript pattern
This example assumes the selected item count increases as more content loads. It collects text as a simple demonstration; for production, deduplicate by a stable ID or URL where available.
async function collectByScrolling(page, {
itemSelector,
scrollTargetSelector = null,
maxRounds = 30,
noProgressLimit = 3,
}) {
let noProgress = 0;
const items = new Set();
for (let round = 0; round < maxRounds && noProgress < noProgressLimit; round++) {
const before = await page.$$eval(itemSelector, els =>
els.map(el => el.textContent?.trim()).filter(Boolean)
);
before.forEach(item => items.add(item));
const previousCount = before.length;
await page.evaluate((selector) => {
const target = selector
? document.querySelector(selector)
: document.scrollingElement;
if (!target) throw new Error('Scroll target not found');
target.scrollTop = target.scrollHeight;
}, scrollTargetSelector);
try {
await page.waitForFunction(
(selector, count) => document.querySelectorAll(selector).length > count,
{ timeout: 5000 },
itemSelector,
previousCount,
);
noProgress = 0;
} catch {
noProgress++;
}
}
return [...items];
}Replace itemSelector and, when needed, scrollTargetSelector with selectors from the target page. The five-second wait is an example budget, not a universal timing recommendation. A selector that is already present is not proof of progress: Puppeteer’s selector wait can return immediately when that selector exists. Compare a changed count, identity, or state instead.
Choose a signal that fits the page
Document versus nested scrolling
When the document scrolls, document.scrollingElement is a reasonable target. When a feed or results panel scrolls independently, scrolling the document may move the surrounding page while leaving the feed untouched. Pass the panel’s selector and verify that its scrollTop changes.
Item count versus page height
An increasing item count is useful when each load appends rendered elements. Document height alone is a weaker signal: nested containers may grow independently, and virtualized lists may recycle rows without changing their count or total document height. Prefer an item identity or another state change that reflects the data you need.
Fixed delays versus observable waits
A fixed sleep can finish before a slow response or waste time when the page is already ready. Prefer a predicate or locator wait tied to the expected change. Use a timeout to bound that wait, then inspect the cause of a timeout rather than increasing delays indefinitely.
Troubleshoot common failures
- Scrolling happens but no new content appears: check whether a nested element owns scrolling, whether the page needs a particular threshold, and whether your action reaches it. Inspect the page for consent, login, or bot-check screens that may prevent the expected feed from appearing.
- The script keeps reading old content: wait for a meaningful transition, such as a higher count or a new item ID, not merely for the scroll command to complete.
- The wait returns immediately: the selector may already exist from earlier content. Wait for a changed count, identity, or loader transition; presence alone does not establish that another batch loaded.
- The loop never ends: add a site-specific end condition, a consecutive no-progress limit, and a maximum round or elapsed-time budget. An infinite feed may not have a meaningful finite bottom.
- Document height stays unchanged: look for a nested scroller, a virtualized list, or a threshold/intersection-based trigger. Track a signal that changes when the desired data becomes available.
- The wait times out: confirm that the expected selector and predicate match the current page, then check for failed or blocked requests and site errors. A timeout means the chosen condition was not observed in time; it does not by itself prove the page has no more results.
Or skip the browser setup
If your goal is a screenshot rather than collecting every feed item, ScreenshotNeo can return a screenshot or PDF from one request, without setting up Puppeteer. For example, this cURL call captures a page:
curl -G "https://api.screenshotneo.com/v1/shot" -d access_key=YOUR_API_KEY --data-urlencode url=https://stripe.com -o shot.webpSee the ScreenshotNeo API documentation for request options. It accepts cookie banners and removes more than 60 known consent platforms, newsletter popups, and chat widgets before capture; each step can be turned off. Bot checks, blank pages, failed loads, timeouts, and cache hits are not billed, and responses indicate the page verdict and billing status. Its MCP server provides take_screenshot, get_page_info, and capture_pdf for AI agents. The Free plan includes 1,000 shots per month with no card; paid plans start at $5 for 3,000 shots.
Sign up for ScreenshotNeo’s free plan to get 1,000 screenshots a month with no card.
Frequently Asked Questions
Does Puppeteer provide a built-in infinite-scroll loop?
No universal loop is prescribed by the documented APIs. Compose page evaluation and condition-based waits around the target site’s scroll behavior and progress signal.
Why is item count unreliable on some infinite feeds?
Virtualized interfaces can recycle existing DOM rows, so the number of rendered elements may not rise when additional results become available. Track stable item identities or a page-specific state change instead.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
