Use a real browser to load the page, wait for the content your target needs, select the relevant elements, and map their text, links, or attributes into structured records. Playwright is a practical default because its locators auto-wait and retry, while page-context evaluation lets you run focused DOM extraction. Always validate counts and required fields before accepting the result.
Choose the simplest source first
Before automating a visible browser, check whether the site offers an API, export, RSS or other structured feed for the data you need. A supported interface is usually easier to maintain and less expensive to run. If the information exists only after JavaScript executes, browser automation can render the page and expose the same DOM a visitor sees.
Confirm the site’s terms and any applicable rules for your project. A robots.txt file concerns crawling instructions for cooperative crawlers; it is not a complete permission or legal analysis.
Install Playwright and create a first extractor
The examples below use Python and synchronous Playwright. Install the package and browser binaries in your project environment:
#1 Best Overall
- Read Before You Buy — No Video Output: These adapters support charging and USB 2.0 data transfer, but cannot transmit video signals. Except for standard USB webcams (which use USB data only), they are not compatible with HDMI/DisplayPort cables, video-capable USB-C hubs, or docking stations with video output.
- Convert USB-A Ports to USB-C: Designed to connect USB-C earphones, cables, flash drives, card readers, and other USB-C accessories to standard USB-A ports. Plug-and-play with no drivers or software required.
- Aluminum Alloy Housing: Built with a sturdy aluminum alloy shell that aids in heat dissipation and protects against daily wear and scratches. Designed to maintain a stable and secure connection.
- Compact & Travel-Friendly: The ultra-compact design allows the adapter to stay plugged into your device without blocking adjacent ports or adding bulk, reducing wear and tear on your original USB ports.
- 12-Month Warranty: Backed by a 12-month manufacturer warranty for peace of mind. Designed to meet strict quality control standards for reliable everyday performance.
python -m pip install playwright
python -m playwright install chromiumThis script opens a page, waits for product cards, extracts explicit fields, and writes JSON. Replace the URL and selectors with those from the page you are permitted to process.
from playwright.sync_api import sync_playwright
import json
URL = "https://example.com/catalog"
with sync_playwright() as p:
browser = p.chromium.launch(headless=True)
page = browser.new_page()
page.goto(URL, wait_until="domcontentloaded", timeout=60_000)
cards = page.locator("article.product-card")
cards.first.wait_for(state="visible", timeout=30_000)
records = cards.evaluate_all("""
nodes => nodes.map(node => ({
name: node.querySelector('.product-name')?.textContent?.trim() || null,
price: node.querySelector('.price')?.textContent?.trim() || null,
url: node.querySelector('a')?.href || null
}))
""")
if not records:
raise RuntimeError("No product cards matched; inspect the selector or page state")
for index, record in enumerate(records, start=1):
if not record["name"] or not record["url"]:
raise ValueError(f"Record {index} is missing a required field: {record}")
print(json.dumps(records, ensure_ascii=False, indent=2))
browser.close()goto() reaching a navigation state does not guarantee that a client-rendered list is ready. Waiting for a meaningful element makes the extraction depend on the page state you actually need rather than on an arbitrary sleep.
Wait for dynamic content deliberately
Wait for a specific element
Use a locator for the list, card, table or field that proves the data has appeared:
page.get_by_role("heading", name="Latest reports").wait_for()
rows = page.locator("table tbody tr")
rows.first.wait_for(state="visible")For a state that changes after an action, wait for the expected result:
Do these 3 things before closing this tab:
1Scan for outdated or missing drivers - takes under a minute2Clear out junk files and repair common Windows errors3Fix the driver behind crashes, sound loss and screen glitchespage.get_by_role("button", name="Load more").click()
page.locator("article.result").nth(20).wait_for(state="visible")A fixed delay can still be useful for a known animation, but it is a poor substitute for a condition. If a list is populated in several batches, wait for a stable count or for a completion marker before collecting all matches.
Understand collection timing
Playwright locators resolve against the current page and provide auto-waiting for actions. A multiple-element query such as locator.all() does not itself wait for a dynamically loaded list to finish. Wait for the page’s completion signal first, then collect the elements.
Rank #2
- 5-in-1 USB-C Hub: Experience comprehensive connectivity featuring a Power Delivery input, two USB-A 2.0 ports, a USB-A 3.0 port, and an HDMI port. (Note: The USB-C power delivery input port is only for connecting an external wall charger to power your laptop and cannot power peripheral devices.)
- 90W Pass-Through Charging: Achieve optimal charging with 90W pass-through power to your laptop, supported by a total input of 100W, with the hub reserving 10W for operational efficiency. (Note: Wall charger not included.)
- Quick Data Transfers: Accelerate your productivity with rapid data transfers using a high-speed 5Gbps USB 3.0 port and two 480Mbps USB 2.0 ports.
- 4K HDMI Display: Enhance your visual experience with a hub capable of delivering 4K resolution at 30Hz in both mirror and extend modes. Please note that this hub is compatible with MacBook (macOS 12 and newer), Windows 10 and 11, ChromeOS, and laptops equipped with DP Alt Mode and Power Delivery. Note: This device is not compatible with Linux.
- What You Get: Anker USB-C Hub (5-in-1, 4K HDMI), welcome guide, 18-month warranty, and our friendly customer service.
items = page.locator(".result")
page.locator("[data-load-complete='true']").wait_for()
for item in items.all():
print(item.inner_text())Select elements that survive redesigns
Prefer user-facing locators
When the target has a meaningful accessible name, use roles, labels or visible text. These describe what a user interacts with and are often less coupled to implementation details:
page.get_by_role("link", name="Download report")
page.get_by_label("Email address")
page.get_by_text("In stock", exact=True)Verify uniqueness. Playwright operations that imply one target are strict; multiple matches can raise an error. Do not add .first() or .nth() merely to hide an ambiguous selector. Narrow it by region, name or relationship so it identifies the intended control.
Crashes, No Sound, or Screen Glitches?
Random freezes, missing sound and display glitches usually trace back to one bad driver. Find and replace yours safely.Free scan · under a minutePC Slower Than It Used to Be?
A free scan shows the junk files, broken settings and background clutter dragging Windows down - then fixes them in one click.Free scan · Windows 10 & 11Use test IDs when they are an explicit contract
If the application exposes a stable test attribute, it can be an excellent extraction hook:
page.get_by_test_id("order-row")Test IDs are implementation-specific and may not exist on a third-party site, so treat them as a deliberate contract rather than a universal solution.
Use CSS or XPath for focused structural work
CSS is useful for concise batch queries and attributes:
links = page.locator("main article a[href]").evaluate_all(
"nodes => nodes.map(a => ({text: a.textContent.trim(), href: a.href}))"
)XPath can express a relationship that is awkward in CSS, but long paths tied to nested containers and generated classes are brittle. Invalid CSS syntax raises an error; unusual IDs or class values may need escaping.
The Tool Desk
Outbyte PC Repair FREERepair Windows errors before they cause bigger problemsFix Now →Outbyte Driver Updater FREEFix the driver behind crashes, sound loss and screen glitchesFind Drivers →Rank #3
- Sleek 7-in-1 USB-C Hub: Features an HDMI port, two USB-A 3.0 ports, and a USB-C data port, each providing 5Gbps transfer speeds. It also includes a USB-C PD input port for charging up to 100W and dual SD and TF card slots, all in a compact design.
- Flawless 4K@60Hz Video with HDMI: Delivers exceptional clarity and smoothness with its 4K@60Hz HDMI port, making it ideal for high-definition presentations and entertainment. (Note: Only the HDMI port supports video projection; the USB-C port is for data transfer only.)
- Double Up on Efficiency: The two USB-A 3.0 ports and a USB-C port support a fast 5Gbps data rate, significantly boosting your transfer speeds and improving productivity.
- Fast and Reliable 85W Charging: Offers high-capacity, speedy charging for laptops up to 85W, so you spend less time tethered to an outlet and more time being productive.
- What You Get: Anker USB-C Hub (7-in-1), welcome guide, 18-month warranty, and our friendly customer service.
Extract text, attributes and structured values
Locator evaluation
Use evaluate() or evaluate_all() for a deliberate transformation in the page context. Return plain serializable objects rather than DOM nodes:
rows = page.locator("table tbody tr").evaluate_all("""
nodes => nodes.map(row => {
const cells = [...row.querySelectorAll('td')];
return {
title: cells[0]?.textContent.trim() || null,
status: cells[1]?.textContent.trim() || null,
href: row.querySelector('a')?.href || null
};
})
""")Keep the browser-side function focused on extraction. Parsing, normalization and validation are usually easier to test in your application code after the values have crossed the browser boundary.
Direct DOM selection with querySelectorAll()
For a simple page-context query:
values = page.evaluate("""
() => [...document.querySelectorAll('ul.features li')]
.map(el => el.textContent.trim())
""")The DOM method returns a static NodeList in document order. It does not update when the page changes. If clicking a filter, loading another page or scrolling changes the content, run the query again after that update.
Handle pagination, lazy loading and interaction
Pagination
Inspect whether the page uses numbered links, a next button, an API-backed cursor or an infinite scroll. Extract one page, record a deduplication key, then follow the next state only when it exists. Stop on a disabled control, a missing next URL or a repeated cursor. Do not assume the number of pages from the first response.
What’s actually slowing this PC down?
Pick the symptom - the matching free tool is one click away.
Lazy-loaded records
Scroll or trigger the site’s own load control, then wait for a newly expected element or completion marker. Capture the count before and after each action; a timeout or unchanged count should be treated as a failure signal, not silently accepted as a complete dataset.
Content hidden behind an interaction
Click the tab, consent control or disclosure that a normal visitor would use, then wait for the target region. Keep the interaction narrow and document it in your extractor so a future UI change is obvious.
Rank #4
- Dual Converters, Infinite Potential：Includes 2× USB C male to USB A female adapters and 2× USB A male to USB C female adapters. Perfect for a wide range of uses—tablets with Bluetooth keyboards, expand USB ports on macbook, and more. Two different converters for all your daily needs
- Next-Level 10Gbps & 3A Charging: No more slow 480Mbps, this usb to usb c adapter has a transfer speed of up to 10Gbps, allowing you to do more transferring in less time. This usb adapter fits both USB A and USB C charger, supporting up to 3A fast charging
- Upgraded Exquisite Craftsmanship: With an aluminum alloy housing and metal connector, the usbc to usb adapter is extremely durable and sturdy. Rigorously tested to withstand more than 10,000 times of plugging and unplugging, ensuring long-lasting performance
- Broad Compatible: The usb c to usb adapter widely supports all USB C/ USB A devices like laptops, tablets, cellphones, car chargers, and phone chargers. Such as compatible with MacBook Pro/Air 2023/2022, Thunderbolt 4/3 Devices,Apple MagSafe Watch 9/8/7/SE/Ultra, iPad Pro 2022/2021, Samsung Galaxy S23/S20/S10, and iPhone 17/16/15 Pro. Plug and play
- Please Note: To reach 10Gbps speed, keep the cable under 3.3 ft. For USB A Male to USB C adapters, try flipping the USB C connector. USB C Male to USB A adapters support bidirectional 10Gbps transfer within 3.3 ft
Validate before saving data
Extraction that returns an empty array can look successful unless you check it. At minimum, validate:
- the expected page or record identifier is present;
- the match count is within a sensible range;
- required fields are non-empty and have the expected type;
- URLs are absolute or can be resolved against the page origin;
- duplicate keys are handled explicitly;
- a representative record matches what is visibly rendered.
Log the URL, selector version, timestamp and count. Keep a small fixture or saved HTML sample for regression tests. A zero-match result should stop the job or enter a review queue instead of producing a plausible-looking empty file.
Quick wins for a faster PC:
Scan for outdated or missing drivers - takes under a minuteDriver Scan →Clear out junk files and repair common Windows errorsFree Scan →Common failures and fixes
“No elements found”
Cause: the selector is wrong, the content is still rendering, or a different route is served to automation. Fix: inspect the rendered DOM, wait for a meaningful element, confirm the URL and check whether an interaction is required.
Timeout while waiting
Cause: the page failed, a selector never appears, or the site takes longer than your limit. Fix: capture a screenshot and console/network logs, raise the timeout only when justified, and fail with the URL and selector included in the error.
Partial list
Cause: pagination, lazy loading or an infinite scroll was not completed. Fix: detect the site’s completion condition and compare counts after each page or scroll.
Ambiguous locator error
Cause: a strict locator matched several nodes. Fix: scope it to the relevant card or region and add a distinguishing role, label, text or test ID. Do not conceal the problem with .first().
Best Value
- 5-in-1 Connectivity: Equipped with a 4K HDMI port, a 5 Gbps USB-C data port, two 5 Gbps USB-A ports, and a USB C 100W PD-IN port. Note: The USB C 100W PD-IN port supports only charging and does not support data transfer devices such as headphones or speakers.
- Powerful Pass-Through Charging: Supports up to 85W pass-through charging so you can power up your laptop while you use the hub. Note: Pass-through charging requires a charger (not included). Note: To achieve full power for iPad, we recommend using a 45W wall charger.
- Transfer Files in Seconds: Move files to and from your laptop at speeds of up to 5 Gbps via the USB-C and USB-A data ports. Note: The USB C 5Gbps Data port does not support video output.
- HD Display: Connect to the HDMI port to stream or mirror content to an external monitor in resolutions of up to 4K@30Hz. Note: The USB-C ports do not support video output.
- What You Get: Anker 332 USB-C Hub (5-in-1), welcome guide, our worry-free 18-month warranty, and friendly customer service.
Stale values after a click
Cause: you queried before the DOM update, or retained a static querySelectorAll() result. Fix: wait for the new state and query again.
Bot check or CAPTCHA
Cause: the site is challenging automated access. Fix: do not attempt to bypass a protection you are not authorized to bypass. Use an approved API, contact the site owner, or stop the job.
Performance, reliability and operating cost
Reuse a browser process when processing many pages, but isolate contexts when cookies, locale or authentication must differ. Limit concurrency to what the target and your network can reasonably handle. Set navigation and action timeouts, retry transient failures with backoff, and record whether a failure occurred during navigation, waiting or extraction. Cache results when freshness permits, and avoid downloading images, fonts or third-party resources that cannot affect the fields you need.
Run a small canary URL after selector changes. Compare record counts and required-field rates with previous runs. A successful HTTP response is not proof that the desired data was rendered; your assertions are the reliability boundary.
Recommended Free Tools
Or skip the browser setup
For a one-off screenshot of a rendered page, or when an AI workflow needs visual page capture rather than DOM records, ScreenshotNeo provides a single request. It accepts consent banners like a visitor and removes more than 60 known consent platforms, newsletter popups and chat widgets before capture; each cleanup step can be disabled. Bot checks, blank pages, timeouts, failed loads and cache hits are not billed, and response headers report the page verdict and billing status. Its MCP server exposes take_screenshot, get_page_info and capture_pdf to Claude, Cursor and other MCP clients.
See the ScreenshotNeo API documentation for options such as full-page lazy-image loading, CSS-selector element capture, custom JavaScript, waits, request blocking, cookies, headers, geolocation, PDFs and bulk jobs.
curl -G "https://api.screenshotneo.com/v1/shot" -d access_key=YOUR_API_KEY --data-urlencode url=https://stripe.com -o shot.webpimport requests
r = requests.get("https://api.screenshotneo.com/v1/shot", params={"access_key": "YOUR_API_KEY", "url": "https://stripe.com"}, timeout=90)
r.raise_for_status()
open("shot.webp", "wb").write(r.content)const q = new URLSearchParams({ access_key: 'YOUR_API_KEY', url: 'https://stripe.com' });
const res = await fetch(`https://api.screenshotneo.com/v1/shot?${q}`);
if (!res.ok) throw new Error(`HTTP ${res.status}`);
const fs = await import('node:fs/promises');
await fs.writeFile('shot.webp', Buffer.from(await res.arrayBuffer()));Every plan includes the features above. The Free plan provides 1,000 shots per month with no card; paid plans start at $5 for 3,000 shots. Create a free ScreenshotNeo account to try it without a card.
FAQ
Should I use browser automation for every scraping job?
No. Prefer an authorized API, export or feed when one supplies the fields you need. Use a browser when the value depends on client-side rendering or interaction.
Free tools Windows power users keep installed
One-click scans. No signup required.
Why did my CSS selector work yesterday and fail today?
Selectors tied to classes or DOM nesting are implementation details. A redesign can change them without changing the visible content. Prefer accessible meaning or an explicit test contract, and keep selector checks in regression tests.
Can I treat robots.txt as permission?
No. Robots directives describe crawler preferences for cooperative crawlers. They do not answer every contractual, copyright, privacy or other legal question for your project.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
