Use a JavaScript-capable browser automation tool, scroll the product list in controlled steps, wait for new cards to render, and save each product under a stable identifier such as its product URL. Stop only when the site signals the end or several scroll-and-wait cycles add no new unique products—and check the result against any total shown on the page. This is a practical collection method, not proof that a store has exposed its entire catalog.
First identify how the listing loads
Stores can reveal more products in different ways, and scrolling works only when the page uses scroll-triggered loading. Google Search Central distinguishes pagination, “Load more,” and infinite scroll as separate ways to display part of a larger ecommerce result set. Google’s ecommerce pagination guidance also explains that its crawler generally does not click buttons or trigger JavaScript that requires user action.
- Infinite scroll: Scrolling near the end of the list requests or reveals another batch.
- Load more: A button explicitly requests the next batch; click it and wait for cards to appear.
- Pagination: Separate pages have next-page links or controls; collect each page rather than expecting one long list.
- Nested list: The product panel has its own scrollbar, so the page itself may not need to move.
Use a browser that executes the site’s JavaScript. A static HTML request may contain only the cards rendered initially and miss products revealed after interaction.
Capture products with Playwright
The following JavaScript pattern records product links, scrolls a verified bottom sentinel into view, and stops after three consecutive rounds with no increase in unique links. Replace the selectors and waiting logic with ones verified on the store. It is an adaptable sketch, not a tested script for a particular site.
#1 Best Overall
const cards = page.locator("YOUR_PRODUCT_CARD_SELECTOR");
const seen = new Set();
let unchangedRounds = 0;
while (unchangedRounds < 3) {
// Read the currently rendered product links.
const ids = await cards.locator("a[href]").evaluateAll(links =>
links.map(a => a.href)
);
const before = seen.size;
for (const id of ids) seen.add(id);
// Use a verified sentinel near the end of the listing.
await page.getByText("YOUR_BOTTOM_SENTINEL").scrollIntoViewIfNeeded();
// Replace this delay with a site-specific loading or change condition.
await page.waitForTimeout(750);
const nextIds = await cards.locator("a[href]").evaluateAll(links =>
links.map(a => a.href)
);
for (const id of nextIds) seen.add(id);
unchangedRounds = seen.size === before ? unchangedRounds + 1 : 0;
}
console.log([...seen]);Make the selectors and identifier site-specific
Inspect the page to find a selector matching product cards and a link or site-specific product ID that uniquely identifies each product. A card may contain several links, so collecting every a[href] beneath every card can capture category links, variants, or promotional links instead of one product identifier. If necessary, narrow the link selector or extract an ID from the card itself.
Normalize URLs only when you know which differences are irrelevant to product identity. For example, removing tracking parameters may help deduplicate links, but removing variant parameters could collapse distinct product listings. Do not treat a repeated card as a new product merely because it was re-rendered.
Wait for a meaningful change
The fixed 750 ms delay is only a simple placeholder. Prefer a condition tied to the page, such as a loading indicator disappearing, a card count increasing, a new product identifier appearing, or a known end-of-results message becoming visible. A delay that is too short can make a slow request look like the end; a long delay on every round can waste time.
Playwright documents scrolling a bottom element into view as a way to trigger an infinite list. For a nested scrollable panel, hover over the verified container and use the mouse wheel, or adjust that container’s scrollTop, rather than scrolling the document. See Playwright’s input documentation.
Do not enumerate a changing list too early
Wait until the relevant batch has appeared before collecting it. Playwright’s locator documentation notes that locator.all() immediately returns the elements currently present and may be unpredictable when the list is changing dynamically. See the locator.all() documentation.
Decide when the capture is complete
Use an explicit stopping check rather than assuming that reaching the current bottom means the listing is finished.
- Record unique product identifiers from the currently rendered cards.
- Scroll the correct target or activate the listing’s control.
- Wait for the page’s loading state or product identifiers to change.
- Record newly rendered identifiers and compare the unique count with the count before the action.
- Continue until the site shows a clear end state, or several successive scroll-and-wait cycles produce no new unique identifiers.
- If the page displays a result total, compare it with the unique identifiers you collected and investigate any difference.
Three unchanged rounds is an operational threshold in the example, not an official Playwright rule or a guarantee. If progress stops unexpectedly, check whether the request is still pending, whether the wrong container is scrolling, or whether the site has switched to a button or page-based pattern. A displayed total can also differ from the number of visible unique cards because of duplicates, hidden or unavailable products, filters, or lazy loading.
When scrolling is not the right method
For a “Load more” control
Click the button, wait for the loading state to finish or for the card count or identifiers to change, then collect the appended cards. Repeat until the control disappears, becomes disabled, or the site presents another clear end state. Scrolling alone may not request another batch.
Free tools Windows power users keep installed
One-click scans. No signup required.
For paginated listings
Collect each page and follow its next-page link or control until there is no next page. Google recommends crawlable sequential links for paginated content; its crawler generally follows URLs in anchor href attributes rather than clicking buttons. This is relevant to indexing, but it does not guarantee that any particular automation workflow can retrieve a store’s full catalog.
Troubleshoot missing or duplicate products
- The page moves but no products appear: Confirm the listing actually uses infinite scroll. Look for a “Load more” control or pagination, and verify that your selector points to a product card rather than an unrelated page element.
- The document does not move: Check for a nested scrollable product panel. Scroll that container directly or use a sentinel inside it. Playwright documents both scrolling an element into view and controlling the wheel or a specific container. See its input guidance.
- The capture stops after the first batch: The script may be reading before the asynchronous request finishes. Wait for a card count or identifier change, a loading indicator to disappear, or another page-specific signal.
- The output count varies between runs: The list may be changing while it is being enumerated. Wait for the current batch to stabilize; avoid relying on
locator.all()as a wait mechanism. Playwright’s locator documentation describes its immediate-return behavior. - The unique count is lower than the site total: Check whether your selector misses cards, filters are active, some products are hidden or unavailable, or the total includes items not currently rendered. Confirm that the deduplication key distinguishes product variants if those are separate listings.
- The script never reaches its stopping condition: A changing banner, rotating promotion, or unstable identifier may keep making the page appear to progress. Track stable product IDs and inspect the page’s end state instead of treating every DOM change as a new product.
Or skip the browser setup
For a screenshot rather than a structured catalog export, ScreenshotNeo is a website screenshot API and MCP server. Its one-request API returns an image or PDF, but a screenshot is not a substitute for extracting each product’s structured details or IDs.
For example, request a screenshot of the listing page as it appears at the supplied URL:
Quick Recap
curl -G "https://api.screenshotneo.com/v1/shot" -d access_key=YOUR_API_KEY --data-urlencode url=https://stripe.com -o shot.webpReplace the example URL with the listing URL. See the ScreenshotNeo API documentation for request options. Cookie banners, popups, and chat widgets are removed before the shot; bot checks, blank pages, and failed loads are never billed. An MCP server lets AI agents take screenshots, and the Free plan includes 1,000 screenshots a month with no card; paid plans start at $5 for 3,000.
PC Slower Than It Used to Be?
A free scan shows the junk files, broken settings and background clutter dragging Windows down - then fixes them in one click.Free scan · Windows 10 & 11Crashes, No Sound, or Screen Glitches?
Random freezes, missing sound and display glitches usually trace back to one bad driver. Find and replace yours safely.Free scan · under a minuteSign up for ScreenshotNeo’s free plan.
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
