The Tool Desk
Outbyte Driver Updater FREEFix the driver behind crashes, sound loss and screen glitchesFind Drivers →Outbyte PC Repair FREERepair Windows errors before they cause bigger problemsFix Now →Scrape a single-page application (SPA) by waiting for the specific UI state that contains the data, then reading the rendered DOM with Playwright locators. domcontentloaded and load are useful navigation milestones, but neither proves that client-side requests and rendering have finished. Treat readiness as an application condition: a result row is visible, a loading indicator disappears, a result count reaches an expected value, or a route change is followed by the target content.
What makes an SPA different to scrape?
An SPA usually delivers a small HTML shell and JavaScript bundle. After that document loads, JavaScript requests JSON or other resources, updates the page, and changes views without a full navigation. A scraper that reads the initial HTML can therefore see an empty root element, a loading message, or only a template.
Playwright controls a real browser, so it can execute that JavaScript and expose the same rendered DOM a user sees. The reliable sequence is:
- Open the intended route.
- Wait for a condition that proves the required data is available.
- Use locators to extract text and attributes.
- For changing routes or lists, synchronize the transition and verify the resulting content.
Check the target site’s terms, robots policy, authentication rules and rate limits before automating access. Browser automation APIs do not grant permission to collect data.
Free tools Windows power users keep installed
One-click scans. No signup required.
#1 Best Overall
- Read Before You Buy — No Video Output: These adapters support charging and USB 2.0 data transfer, but cannot transmit video signals. Except for standard USB webcams (which use USB data only), they are not compatible with HDMI/DisplayPort cables, video-capable USB-C hubs, or docking stations with video output.
- Convert USB-A Ports to USB-C: Designed to connect USB-C earphones, cables, flash drives, card readers, and other USB-C accessories to standard USB-A ports. Plug-and-play with no drivers or software required.
- Aluminum Alloy Housing: Built with a sturdy aluminum alloy shell that aids in heat dissipation and protects against daily wear and scratches. Designed to maintain a stable and secure connection.
- Compact & Travel-Friendly: The ultra-compact design allows the adapter to stay plugged into your device without blocking adjacent ports or adding bulk, reducing wear and tear on your original USB ports.
- 12-Month Warranty: Backed by a 12-month manufacturer warranty for peace of mind. Designed to meet strict quality control standards for reliable everyday performance.
Install Playwright and create a browser context
In a Node.js project, install Playwright and its browser binaries:
npm install playwright
npx playwright install chromiumThis complete example uses Chromium in headless mode. Replace the URL and selectors with those from the application you are allowed to access.
const { chromium } = require('playwright');
(async () => {
const browser = await chromium.launch({ headless: true });
const context = await browser.newContext();
const page = await context.newPage();
try {
await page.goto('https://example.com/app', {
waitUntil: 'domcontentloaded',
timeout: 30_000
});
const results = page.locator('[data-testid="result"]');
await results.first().waitFor({ state: 'visible', timeout: 30_000 });
const rows = await results.evaluateAll(nodes =>
nodes.map(node => ({
title: node.querySelector('[data-testid="title"]')?.textContent?.trim() || '',
href: node.querySelector('a')?.href || ''
}))
);
console.log(JSON.stringify(rows, null, 2));
} finally {
await browser.close();
}
})();waitUntil: 'domcontentloaded' only waits until the document has been parsed. You could choose load when the page’s load event is a useful initial milestone, but still add an application-specific wait afterward.
Wait for the rendered content, not an arbitrary delay
Use a locator for a meaningful element
Locators are retryable and auto-waiting. They resolve against the current page state, which matters when a framework replaces nodes during rendering. Prefer a stable selector such as a data attribute, semantic role or distinctive heading:
const pageTitle = page.getByRole('heading', { name: 'Orders' });
await pageTitle.waitFor({ state: 'visible' });
const title = await pageTitle.textContent();If the application displays a loading indicator, wait for it to disappear and for the content to appear. Waiting for both avoids extracting a partially rendered view:
await page.locator('[data-testid="loading"]').waitFor({ state: 'hidden' });
await page.locator('[data-testid="order-row"]').first().waitFor({ state: 'visible' });Wait for a status or count to change
Some pages have a stable container before data arrives. In that case, assert the state that proves completion:
const status = page.locator('[role="status"]');
await status.waitFor({ state: 'visible' });
await expect(status).toHaveText(/d+ results/);
const count = page.locator('[data-testid="result-count"]');
await expect(count).not.toHaveText('Loading…');If you use assertions, import expect from Playwright’s test package or implement an equivalent polling loop in a standalone script. Set a timeout that matches the site’s normal response time and your deployment environment.
Rank #2
- 5-in-1 USB-C Hub: Experience comprehensive connectivity featuring a Power Delivery input, two USB-A 2.0 ports, a USB-A 3.0 port, and an HDMI port. (Note: The USB-C power delivery input port is only for connecting an external wall charger to power your laptop and cannot power peripheral devices.)
- 90W Pass-Through Charging: Achieve optimal charging with 90W pass-through power to your laptop, supported by a total input of 100W, with the hub reserving 10W for operational efficiency. (Note: Wall charger not included.)
- Quick Data Transfers: Accelerate your productivity with rapid data transfers using a high-speed 5Gbps USB 3.0 port and two 480Mbps USB 2.0 ports.
- 4K HDMI Display: Enhance your visual experience with a hub capable of delivering 4K resolution at 30Hz in both mirror and extend modes. Please note that this hub is compatible with MacBook (macOS 12 and newer), Windows 10 and 11, ChromeOS, and laptops equipped with DP Alt Mode and Power Delivery. Note: This device is not compatible with Linux.
- What You Get: Anker USB-C Hub (5-in-1, 4K HDMI), welcome guide, 18-month warranty, and our friendly customer service.
Wait for URL transitions, then verify the view
A click can change the SPA route without loading a new document. Synchronize the URL and the content together:
await Promise.all([
page.waitForURL('**/reports/**'),
page.getByRole('link', { name: 'Reports' }).click()
]);
await page.getByRole('heading', { name: 'Reports' }).waitFor({ state: 'visible' });The URL match confirms routing, not data readiness. Always add the locator or state check needed by your extraction.
Why fixed sleeps are fragile
page.waitForTimeout(2000) may be too short on a slow run and waste time on a fast one. It also says nothing about whether the requested records, an error message or an empty state has appeared. Use a timeout only as a safety boundary around an observable condition.
Should you use networkidle?
Playwright defines networkidle as no network connections for at least 500 ms and labels it discouraged as a general readiness signal. An SPA may keep analytics, polling or streaming connections open, so the event may never occur. Conversely, a quiet network can occur before the UI has processed the response.
Use networkidle only when you understand the application’s behavior and it is an intentional part of your synchronization. For most scrapers, navigate with domcontentloaded or load, then wait for the result-specific locator or state.
Quick wins for a faster PC:
Clear out junk files and repair common Windows errorsFree Scan →Fix the driver behind crashes, sound loss and screen glitchesFind Drivers →Repair Windows errors before they cause bigger problemsFix Now →| Strategy | Observes | Limitation | Use it when |
|---|---|---|---|
domcontentloaded or load |
Document lifecycle | Client-side fetching and rendering may continue | Choosing an initial navigation milestone |
networkidle |
No network connections for 500 ms | Quiet network does not prove useful content is ready; general use is discouraged | Only when the target’s network behavior makes it meaningful |
| Locator or page-state condition | The element or state your extraction needs | You must identify a meaningful condition | Normal SPA scraping; preferred approach |
| URL wait | A matching main-frame URL | Route change does not prove rendering is complete | Synchronizing navigation, followed by a content check |
Extract text, attributes and structured data
Locator-based extraction
For ordinary fields, keep extraction in locator APIs so Playwright can target the current DOM:
const cards = page.locator('[data-testid="product-card"]');
await cards.first().waitFor({ state: 'visible' });
const total = await cards.count();
const products = [];
for (let i = 0; i < total; i++) {
const card = cards.nth(i);
products.push({
name: (await card.getByRole('heading').innerText()).trim(),
price: (await card.locator('[data-testid="price"]').innerText()).trim(),
url: await card.locator('a').getAttribute('href')
});
}Dynamic lists and locator.all()
locator.all() returns elements present immediately; it does not wait for a dynamic list to finish populating. Wait for a useful stable condition first, then collect:
Rank #3
- Sleek 7-in-1 USB-C Hub: Features an HDMI port, two USB-A 3.0 ports, and a USB-C data port, each providing 5Gbps transfer speeds. It also includes a USB-C PD input port for charging up to 100W and dual SD and TF card slots, all in a compact design.
- Flawless 4K@60Hz Video with HDMI: Delivers exceptional clarity and smoothness with its 4K@60Hz HDMI port, making it ideal for high-definition presentations and entertainment. (Note: Only the HDMI port supports video projection; the USB-C port is for data transfer only.)
- Double Up on Efficiency: The two USB-A 3.0 ports and a USB-C port support a fast 5Gbps data rate, significantly boosting your transfer speeds and improving productivity.
- Fast and Reliable 85W Charging: Offers high-capacity, speedy charging for laptops up to 85W, so you spend less time tethered to an outlet and more time being productive.
- What You Get: Anker USB-C Hub (7-in-1), welcome guide, 18-month warranty, and our friendly customer service.
const items = page.locator('[data-testid="item"]');
await page.locator('[data-testid="list-ready"]').waitFor({ state: 'visible' });
const itemLocators = await items.all();
const values = [];
for (const item of itemLocators) values.push(await item.innerText());If the list grows in batches, wait for a count or an explicit “all results loaded” marker. If no such marker exists, define a domain rule, such as two identical counts separated by a short polling interval, and handle an empty state separately.
Use page evaluation for browser-side processing
page.evaluate() executes in the browser page context, not in the Node.js script context. Browser globals such as document are available there; values from your script must be passed as arguments or serialized:
const rows = await page.evaluate((selector) => {
return [...document.querySelectorAll(selector)].map(row => ({
text: row.textContent.trim(),
href: row.querySelector('a')?.href || null
}));
}, '[data-testid="result"]');Returned promises are awaited. Keep page functions small and return plain serializable data rather than DOM nodes.
Handle pagination, infinite scroll and filters
Paginated routes
For a “Next” button, wait for the old page state to change before extracting the new page:
const firstKey = await page.locator('[data-testid="result"]').first().getAttribute('data-id');
await Promise.all([
page.waitForURL('**/page=2'),
page.getByRole('button', { name: 'Next' }).click()
]);
await page.locator('[data-testid="result"]').first().waitFor({ state: 'visible' });
const secondKey = await page.locator('[data-testid="result"]').first().getAttribute('data-id');When the URL does not change, wait for a page indicator, changed first-record key or updated result count instead of relying on a route event.
Infinite scrolling
Scroll only as far as needed and wait for a measurable increase:
PC Slower Than It Used to Be?
A free scan shows the junk files, broken settings and background clutter dragging Windows down - then fixes them in one click.Free scan · Windows 10 & 11Crashes, No Sound, or Screen Glitches?
Random freezes, missing sound and display glitches usually trace back to one bad driver. Find and replace yours safely.Free scan · under a minuteconst list = page.locator('[data-testid="result"]');
let previous = await list.count();
for (let i = 0; i < 20; i++) {
await page.evaluate(() => window.scrollTo(0, document.body.scrollHeight));
await page.waitForFunction(oldCount =>
document.querySelectorAll('[data-testid="result"]').length > oldCount ||
document.querySelector('[data-testid="end-of-results"]'), previous);
const current = await list.count();
if (current === previous) break;
previous = current;
}Cap iterations, detect an end marker, and preserve a unique ID so retries do not create duplicate records.
Rank #4
- Dual Converters, Infinite Potential：Includes 2× USB C male to USB A female adapters and 2× USB A male to USB C female adapters. Perfect for a wide range of uses—tablets with Bluetooth keyboards, expand USB ports on macbook, and more. Two different converters for all your daily needs
- Next-Level 10Gbps & 3A Charging: No more slow 480Mbps, this usb to usb c adapter has a transfer speed of up to 10Gbps, allowing you to do more transferring in less time. This usb adapter fits both USB A and USB C charger, supporting up to 3A fast charging
- Upgraded Exquisite Craftsmanship: With an aluminum alloy housing and metal connector, the usbc to usb adapter is extremely durable and sturdy. Rigorously tested to withstand more than 10,000 times of plugging and unplugging, ensuring long-lasting performance
- Broad Compatible: The usb c to usb adapter widely supports all USB C/ USB A devices like laptops, tablets, cellphones, car chargers, and phone chargers. Such as compatible with MacBook Pro/Air 2023/2022, Thunderbolt 4/3 Devices,Apple MagSafe Watch 9/8/7/SE/Ultra, iPad Pro 2022/2021, Samsung Galaxy S23/S20/S10, and iPhone 17/16/15 Pro. Plug and play
- Please Note: To reach 10Gbps speed, keep the cable under 3.3 ft. For USB A Male to USB C adapters, try flipping the USB C connector. USB C Male to USB A adapters support bidirectional 10Gbps transfer within 3.3 ft
Filters and search forms
After filling a filter, wait for the result state to reflect the submitted value and for the list to refresh:
await page.getByRole('textbox', { name: 'Search' }).fill('sprocket');
await page.getByRole('button', { name: 'Apply' }).click();
await expect(page.locator('[data-testid="active-query"]')).toHaveText('sprocket');
await page.locator('[data-testid="result"]').first().waitFor({ state: 'visible' });Make runs reliable
Use stable selectors
- Prefer
data-testid, accessible roles and labels over generated class names. - Scope locators to a component to avoid matching duplicate text elsewhere.
- Use exact names or regular expressions when labels vary predictably.
Separate navigation, readiness and extraction
Keep those phases distinct so a timeout identifies the failed assumption. Capture the URL, status text and a screenshot or HTML snapshot on failure. Do not silently treat an empty list as success; distinguish “zero valid records” from “data never loaded.”
Control timeouts and retries
Set navigation and condition timeouts explicitly. Retry transient browser or network failures with a bounded policy, but do not blindly retry selector failures: a changed application can make the selector permanently wrong. On retry, create a fresh page or context when cookies, service workers or in-page state may be contaminated.
Authentication and session state
Use a dedicated context for each account or job. Store credentials securely, avoid logging tokens, and save authenticated storage state only where your security policy permits. Handle redirects to login as a distinct failure condition.
Independent reader supportYour contribution helps us test, update, and keep practical guides available for everyone.Troubleshooting empty or incorrect results
| Symptom | Likely cause | Fix |
|---|---|---|
| Root element is empty | Extraction ran before client rendering | Wait for a result locator, status change or other application condition. |
| Timeout waiting for a result | Wrong selector, blocked request, login redirect or genuine empty state | Inspect the rendered URL and status, test the empty-state selector, and verify access and selectors. |
networkidle never finishes |
Polling, analytics or open connections | Replace it with a content-specific locator or state check. |
| Only the first batch is scraped | List is still loading when all() runs |
Wait for a count, end marker or stable batch before collecting. |
| Old records appear after clicking a filter | SPA updated the URL or controls before replacing the list | Wait for the filter state and a changed result key/count. |
| Works locally, fails in CI | Different timing, viewport, fonts, browser or permissions | Pin browser versions, set a viewport, use explicit timeouts, and retain failure artifacts. |
| Text is missing but visible in the browser | Content is inside an iframe, shadow DOM or canvas | Target the frame, inspect the component’s accessible DOM, or use the application’s data endpoint only when permitted. |
Performance, cost and data quality
Reuse a browser process when running many pages, but isolate jobs with contexts. Limit concurrency to what the target and your network can handle; more tabs can increase memory use and trigger rate limits. Block unneeded resources only after confirming that the application does not require them for rendering. Cache already processed URLs and persist checkpoints so a failed batch resumes without duplication.
Normalize whitespace, parse numbers with locale awareness, preserve the source URL and capture timestamp, and validate required fields before writing. Record explicit empty results and errors separately. A successful HTTP response is not proof that the desired records were present.
Playwright’s API surface and documentation labels can change, especially on pages marked as next-version. Confirm the examples against the Playwright version pinned in your project.
Best Value
- 5-in-1 Connectivity: Equipped with a 4K HDMI port, a 5 Gbps USB-C data port, two 5 Gbps USB-A ports, and a USB C 100W PD-IN port. Note: The USB C 100W PD-IN port supports only charging and does not support data transfer devices such as headphones or speakers.
- Powerful Pass-Through Charging: Supports up to 85W pass-through charging so you can power up your laptop while you use the hub. Note: Pass-through charging requires a charger (not included). Note: To achieve full power for iPad, we recommend using a 45W wall charger.
- Transfer Files in Seconds: Move files to and from your laptop at speeds of up to 5 Gbps via the USB-C and USB-A data ports. Note: The USB C 5Gbps Data port does not support video output.
- HD Display: Connect to the HDMI port to stream or mirror content to an external monitor in resolutions of up to 4K@30Hz. Note: The USB-C ports do not support video output.
- What You Get: Anker 332 USB-C Hub (5-in-1), welcome guide, our worry-free 18-month warranty, and friendly customer service.
Or skip the browser setup
If you need an image or PDF of a rendered page rather than structured records, ScreenshotNeo provides a single-request website screenshot API and MCP server. It accepts cookie and consent banners as a visitor and removes more than 60 known consent platforms, newsletter popups and chat widgets before capture; each step can be disabled. Bot checks, CAPTCHAs, blank pages, timeouts, failed loads and cache hits are not billed, and response headers identify the page verdict and billing status. Its MCP tools—take_screenshot, get_page_info and capture_pdf—work with Claude, Cursor and other MCP clients.
Example request (the ScreenshotNeo API documentation lists all options):
curl -G "https://api.screenshotneo.com/v1/shot" -d access_key=YOUR_API_KEY --data-urlencode url=https://stripe.com -o shot.webpThe same endpoint supports PNG, JPEG or WebP output and options such as full-page capture with lazy images loaded, CSS-selector element capture, dark mode, device presets or custom viewports, retina scale, PDF paper size and page ranges, custom CSS or JavaScript, clicks, selector hiding, waits for selectors or network idle, request blocking, headers, cookies, user agents, authorization, timezone, geolocation, transparent backgrounds, resizing, chosen cache TTLs, signed image links, asynchronous webhooks, bulk capture for up to 100 URLs per call, usage reporting and an OpenAPI specification. Parameter names used by other screenshot APIs also work for easier migration.
Pricing is Free for 1,000 shots per month with no card; Starter is $5 for 3,000; Growth $15 for 15,000; Pro $39 for 60,000; Scale $99 for 250,000; and Business $249 for 1,000,000. Yearly billing provides two months free, and every feature is included on every plan. Sign up free to get 1,000 screenshots a month without a card.
Recommended Free Tools
Frequently Asked Questions
Can Playwright scrape data that never appears in the DOM?
Not directly. If the value is painted only on a canvas or kept inside an inaccessible component, identify a permitted structured endpoint or an application-supported export instead of assuming ordinary DOM locators can read it.
How should I prove that an SPA returned zero results?
Wait for the application’s explicit empty-state indicator or a completed status with a zero count. Do not infer zero from a list that might still be loading.
Is browser automation allowed on every SPA?
No. Review the site’s terms, access controls, robots guidance and applicable law, and honor authentication and rate-limit requirements.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
Do these 3 things before closing this tab:
1Clear out junk files and repair common Windows errors2Fix the driver behind crashes, sound loss and screen glitches3Repair Windows errors before they cause bigger problems
