Do these 3 things before closing this tab:
1Repair Windows errors before they cause bigger problems2Fix the driver behind crashes, sound loss and screen glitches3Clear out junk files and repair common Windows errorsUse Playwright to load a page in a real browser, wait for the content you need, and extract it from the rendered DOM—or capture the network response that supplies it. The reliable approach is to synchronize on a specific locator or response, not to guess with a fixed delay. This guide walks through setup, extraction, dynamic data, selectors, sessions, network control, and common failures.
What Playwright does—and what scraping means here
A conventional HTTP request retrieves the server’s response; it does not necessarily run the JavaScript that fills in a page afterward. Playwright launches a browser, navigates to the page, and lets you inspect its rendered DOM and network traffic. That makes it useful when the information you need appears only after scripts run, a button is clicked, or an API request completes.
There are two main ways to collect page data:
- DOM extraction: Read text or attributes from elements that appear in the rendered page. This follows the content a visitor can see and is often the most direct approach.
- Response extraction: Observe or wait for the API response that supplies the content, then parse its structured data. This can be simpler than reading a complex DOM when the site actually exposes the information in a response.
Playwright is an automation library, not permission to collect any data from any site. Before scraping a target, check its robots.txt, terms of service, authentication requirements, rate limits, copyright and privacy obligations, and the laws that apply to your use. A site-specific policy was not assessed for this general guide.
Install Playwright and its browser
- Install the Node.js package in your project:
npm install playwright. - Install the browser binaries:
npx playwright install. To install a particular browser, usenpx playwright install chromium,npx playwright install firefox, ornpx playwright install webkit. - Save the script below as
scrape.mjs, then run it withnode scrape.mjs.
The library and browser binaries are separate parts of setup: installing the package alone may leave the browser unavailable. The example uses Chromium. Playwright’s normal flow is to launch a browser, create a browser context and page, perform the work, then close the context and browser.
What’s actually slowing this PC down?
Pick the symptom - the matching free tool is one click away.
#1 Best Overall
A complete JavaScript example
This script visits the public Example Domain page, reads its heading and paragraph, and collects its links. The role-based heading locator avoids depending on the page’s specific HTML tag structure. Replace the URL and the data selectors with ones that match a target you are allowed to access.
import { chromium } from 'playwright';
const url = 'https://example.com';
const browser = await chromium.launch();
const context = await browser.newContext();
try {
const page = await context.newPage();
await page.goto(url);
const heading = await page.getByRole('heading').first().textContent();
const paragraph = await page.locator('p').first().textContent();
const links = await page.getByRole('link').evaluateAll((items) =>
items.map((item) => ({
text: item.textContent?.trim() ?? '',
href: item.href,
}))
);
console.log({
url: page.url(),
title: await page.title(),
heading: heading?.trim() ?? null,
paragraph: paragraph?.trim() ?? null,
links,
});
} finally {
await context.close();
await browser.close();
}The paragraph query uses CSS because it is a compact example of selecting a known element type. On a site with stable accessible labels, roles, or test IDs, prefer those contracts instead. Avoid selectors based on incidental nesting, generated class names, or element positions: a harmless redesign can make them point somewhere else or stop matching.
Choose selectors that survive page changes
Playwright locators are the central part of its auto-waiting and retry behavior. Prefer selectors that describe an element’s meaning or a stable site contract, in roughly this order:
getByRole()for accessible roles such as headings, links, buttons, and table rows. Use a name where it distinguishes the target, for examplepage.getByRole('button', { name: 'Load products' }).getByLabel()for form controls connected to a visible label.getByText()when visible text is a reliable way to identify the content.getByPlaceholder(),getByAltText(), andgetByTitle()when those attributes describe the target consistently.getByTestId()when the site deliberately provides test IDs as a stable selector contract.- CSS or XPath when the target has no usable semantic contract or the DOM structure itself is the requirement. Keep these selectors as specific and simple as possible.
A locator can match multiple elements. Use a distinguishing accessible name or a deliberate narrowing step; use .first() or .nth() only when position is genuinely meaningful. Otherwise, selecting the first match can silently extract the wrong item after the page changes. Prefer an explicit count or a uniqueness check when the scraper depends on exactly one result.
Windows Errors? Fix Them Before They Spread
Repair common Windows errors and clear accumulated junk for a smoother, more stable PC - no reinstall needed.Free scan · no reinstallCrashes, No Sound, or Screen Glitches?
Random freezes, missing sound and display glitches usually trace back to one bad driver. Find and replace yours safely.Free scan · under a minuteWait for the content you intend to scrape
page.goto(url) waits for the page’s load event by default. That does not guarantee that every later API call, animation, or application-specific rendering step has finished. Interactions also auto-wait for actionability checks, but a scraper should still wait for the specific content or response it needs before reading it.
Wait for a rendered element
When the page inserts a known element after loading, wait on a locator state and then read it. For example:
const productName = page.getByRole('heading', { name: 'Example product' });
await productName.waitFor({ state: 'visible' });
const name = await productName.textContent();For a dynamic list, wait for a representative row or item, then collect the matching locators. A visible element is usually a stronger signal than an arbitrary sleep, which may be too short on a slow run and unnecessarily long on a fast one.
Wait for the API response that a user action triggers
If clicking a control loads data, create the response promise before clicking. Otherwise, a fast response could arrive before the script begins waiting.
const responsePromise = page.waitForResponse('**/api/products');
await page.getByRole('button', { name: 'Load products' }).click();
const response = await responsePromise;
if (!response.ok()) {
throw new Error(`Products request failed: ${response.status()}`);
}
const data = await response.json();
console.log(data);Use a URL pattern or predicate that identifies the intended endpoint; broad matching can capture an unrelated request. If the endpoint returns something other than JSON, use the corresponding response method, such as text(). Treat observed response shapes as target-specific: an API’s fields and behavior can change independently of the page.
Do not use a fixed delay as your main synchronization strategy
A timeout can be useful when a target has a known, unavoidable pause, but it does not confirm that the data arrived. Generic networkidle waiting and page.waitForSelector are discouraged in Playwright’s testing guidance; locator waits and response synchronization state what the script is actually waiting for. Pages with analytics, polling, or persistent connections can also make network quietness a poor signal. Choose the condition tied to the extraction task.
Observe and control network requests
Use page events when you need to inspect traffic as it occurs. Register listeners before navigating or taking the action that starts the requests:
page.on('request', (request) => {
if (request.url().includes('/api/')) {
console.log('Request:', request.method(), request.url());
}
});
page.on('response', (response) => {
if (response.url().includes('/api/')) {
console.log('Response:', response.status(), response.url());
}
});For a particular action-response pair, waitForResponse() is usually easier to reason about than logging every response. Event listeners are useful for discovery and diagnostics; remove or narrow them in production so unrelated traffic does not overwhelm logs.
Use page.route() or browserContext.route() when you need to intercept matching requests. A route handler must decide what happens to each intercepted request: continue it, fulfill it with a response, or abort it. Routing can help inspect or modify requests, mock a known endpoint, or avoid downloading resources such as images when they are unnecessary to the data you need. Be cautious about blocking scripts, stylesheets, or other resources the application needs to render correctly. A routing rule affects matching URLs, so test its scope before relying on it.
Use browser contexts for independent sessions
A browser context is an isolated, non-persistent session. Cookies belong to the context, and non-persistent contexts do not write browsing data to disk. This makes separate contexts useful when two jobs need independent cookies or permissions, or when one run must not inherit another run’s session. Close each context when the job is done; closing the browser also ends its pages.
const firstContext = await browser.newContext();
const secondContext = await browser.newContext();
try {
const firstPage = await firstContext.newPage();
const secondPage = await secondContext.newPage();
// Run tasks with separate session state.
} finally {
await firstContext.close();
await secondContext.close();
}Keep session state intentional. If a target requires authentication, use only an account and access method you are authorized to use, and follow the target’s terms and applicable rules. Do not assume a fresh context is logged in or that a session from one context is available in another.
Handle pages that use WebSockets
Some applications send updates over WebSockets rather than issuing a conventional request for each change. Playwright exposes a page.on('websocket') event; from the WebSocket object, inspect sent and received frames when that is relevant to the data flow. This is a more specialized route than DOM or HTTP-response extraction. Use it only when the content cannot be reliably obtained through the rendered page or an ordinary response, and keep the extraction tied to the particular connection and message format the target uses.
Make scraping runs more efficient and dependable
- Extract only the fields you need. Read a locator or parse a response rather than serializing an entire page when a few values suffice.
- Avoid unnecessary resources carefully. Routing can abort images or other requests, but blocking dependencies can change rendering. Compare the result with an unrestricted load before making a block rule part of a job.
- Use one context per independent session. This avoids accidental cookie sharing; close contexts and browsers in cleanup paths, including when extraction throws.
- Prefer response data when it is the right source. It can avoid brittle DOM traversal, but an API response is not automatically a stable or documented interface.
- Fail visibly when assumptions break. Check response status, verify expected elements or fields, and log enough context—such as the URL and status—to diagnose an empty or changed result.
- Respect target limits. Keep request volume and frequency within the site’s stated limits and applicable rules. Do not treat browser automation as a way around access controls.
Playwright requires a Node.js package and browser binaries, so execution time and resource use depend on the browser, target page, network, and work performed. The documentation reviewed does not establish a universal runtime or cost figure. For a long-running scraper, measure your own workload and handle timeouts, retries, and partial results deliberately rather than assuming every page will load identically.
Troubleshooting common failures
| Symptom | Likely cause | What to check or change |
|---|---|---|
| Browser launch fails or the executable is missing | The package was installed without the matching browser binary, or the chosen browser was not installed. | Run npx playwright install, or install the specific browser used by the script. Check that the script launches the browser you installed. |
| Navigation returns an error or times out | The URL is wrong, the page is slow or unavailable, or the browser cannot reach it. | Verify the URL and network access. Inspect the navigation error and target response; do not assume increasing a timeout will fix a permanently failing page. |
| Text is empty even though the page opens | The data is rendered later, appears only after an interaction, or the locator does not match. | Inspect the rendered page, wait for the relevant locator, and check whether a click or other user action is required. If an API supplies the data, wait for that response. |
| A locator times out or matches the wrong item | The locator is too broad, based on a changed DOM detail, or does not uniquely identify the target. | Prefer a role, label, text, or test ID; narrow by a stable name or containing region and verify the match count before extraction. |
| The response promise never resolves | The request did not happen, the action did not trigger it, or the URL pattern does not match. | Confirm that the promise is created before the action, inspect requests and responses, and tighten or correct the response predicate. |
| JSON parsing throws | The response was an error, empty, or not JSON. | Check response.ok() and the status first; inspect the response body or use text() if the endpoint does not return JSON. |
| Content differs across runs | The page depends on session state, geography, timing, or changing server data. | Use a deliberate context per session, synchronize on the actual content, and record the conditions relevant to interpreting the result. |
Or skip the browser setup
If your goal is a visual capture rather than structured data extraction, ScreenshotNeo is a website screenshot API: a single GET request can return a PNG, JPEG, WebP, or PDF. It is not a replacement for Playwright when you need to parse text or records. Unlike a DIY browser script, ScreenshotNeo can accept cookie or consent banners as a visitor and remove more than 60 known consent platforms, newsletter popups, and chat widgets before the capture; each step can be turned off. Bot checks and CAPTCHAs, blank pages, timeouts, failed loads, and cache hits are not billed, and response headers report the page verdict and billing status. Its MCP server provides take_screenshot, get_page_info, and capture_pdf tools for Claude, Cursor, and other MCP clients.
Here is the one-call cURL example; replace the URL with the page you need and set your API key. See the ScreenshotNeo API documentation for request options.
curl -G "https://api.screenshotneo.com/v1/shot" -d access_key=YOUR_API_KEY --data-urlencode url=https://stripe.com -o shot.webpFree includes 1,000 screenshots per month with no card; paid plans start at $5 for 3,000 screenshots. Learn about ScreenshotNeo, or sign up free for 1,000 screenshots a month with no card.
Frequently Asked Questions
Does a Playwright navigation timeout prove the target site is down?
No. It only means the awaited navigation condition did not complete within the configured time; network access, page behavior, or the selected wait condition may be responsible. Inspect the error and the page’s requests before concluding the site is unavailable.
Can I scrape data from a WebSocket without reading the page?
Playwright can expose WebSocket connections and their sent and received frames, but interpreting those messages depends on the target’s implementation. First determine whether the same information is available more simply in rendered DOM content or an HTTP response.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
