Start by checking what the website sends over HTTP. If the desired records are already in the HTML or a JSON response, fetch and parse that response with Python; it is usually simpler than running a browser. Use Playwright or Selenium when the data depends on browser rendering or interaction, or when reproducing the underlying request is impractical. A page that looks empty to a basic scraper is a clue to investigate—not proof that a browser is required.
What “dynamic” means for a Python scraper
A web page can show data that was not present in its initial HTML response. JavaScript may request records from a separate endpoint after the page loads, then render them into the page. Content can also appear only after scrolling, clicking, or waiting for a particular state. These cases call for different solutions.
The key question is where the information comes from. Inspect the initial response and the browser’s network activity before choosing a tool. Scrapy’s guidance recommends finding and extracting from the request that contains the desired data when a page fetches it separately: Selecting dynamically-loaded content.
- Data in the initial HTML: an HTTP client and an HTML parser may be enough.
- Data in a separate JSON or HTML request: reproduce that request, if permitted, and parse its response.
- Data that requires rendering or interaction: automate a browser and wait for a specific readiness condition.
How to diagnose an empty or incomplete result
1. Inspect the direct HTTP response
Make a request to the page and examine its status, headers, and body. A successful status does not mean the response contains the records you see in a browser. Search the body for a distinctive visible value, field name, or text fragment. If it is absent, inspect the page’s data requests rather than repeatedly changing HTML selectors.
#1 Best Overall
- Read Before You Buy — No Video Output: These adapters support charging and USB 2.0 data transfer, but cannot transmit video signals. Except for standard USB webcams (which use USB data only), they are not compatible with HDMI/DisplayPort cables, video-capable USB-C hubs, or docking stations with video output.
- Convert USB-A Ports to USB-C: Designed to connect USB-C earphones, cables, flash drives, card readers, and other USB-C accessories to standard USB-A ports. Plug-and-play with no drivers or software required.
- Aluminum Alloy Housing: Built with a sturdy aluminum alloy shell that aids in heat dissipation and protects against daily wear and scratches. Designed to maintain a stable and secure connection.
- Compact & Travel-Friendly: The ultra-compact design allows the adapter to stay plugged into your device without blocking adjacent ports or adding bulk, reducing wear and tear on your original USB ports.
- 12-Month Warranty: Backed by a 12-month manufacturer warranty for peace of mind. Designed to meet strict quality control standards for reliable everyday performance.
from urllib.request import Request, urlopen
url = "https://example.com/catalog"
request = Request(url, headers={"User-Agent": "ExampleResearchBot/1.0"})
with urlopen(request, timeout=20) as response:
body = response.read().decode("utf-8", errors="replace")
print("Status:", response.status)
print("Content type:", response.headers.get("Content-Type"))
print(body[:2000])Replace the example URL and use an honest, appropriate user agent. This inspection code does not bypass access controls, and it does not establish that collection is permitted.
2. Find the request that supplies the visible data
Open the target page in a browser, open its developer tools, select the Network panel, and reload. Filter for Fetch/XHR requests where available, then inspect responses that contain the records you want. Note the request method, URL, query parameters, body, and headers that appear necessary. Some sites return HTML fragments rather than JSON.
Do not assume a URL you happen to find is a stable public API. Recheck the site’s terms and applicable restrictions, and avoid requests that would defeat a login, CAPTCHA, or other access control. Network behavior and permitted use vary by site.
3. Reproduce only the necessary request and parse its format
For a JSON endpoint, keep fetching separate from extraction so you can validate each independently. This example shows the shape of the code; replace the endpoint and keys with the ones you observed and are allowed to use.
The Tool Desk
Outbyte PC Repair FREERepair Windows errors before they cause bigger problemsFix Now →Outbyte Driver Updater FREEFix the driver behind crashes, sound loss and screen glitchesFind Drivers →Rank #2
- 5-in-1 USB-C Hub: Experience comprehensive connectivity featuring a Power Delivery input, two USB-A 2.0 ports, a USB-A 3.0 port, and an HDMI port. (Note: The USB-C power delivery input port is only for connecting an external wall charger to power your laptop and cannot power peripheral devices.)
- 90W Pass-Through Charging: Achieve optimal charging with 90W pass-through power to your laptop, supported by a total input of 100W, with the hub reserving 10W for operational efficiency. (Note: Wall charger not included.)
- Quick Data Transfers: Accelerate your productivity with rapid data transfers using a high-speed 5Gbps USB 3.0 port and two 480Mbps USB 2.0 ports.
- 4K HDMI Display: Enhance your visual experience with a hub capable of delivering 4K resolution at 30Hz in both mirror and extend modes. Please note that this hub is compatible with MacBook (macOS 12 and newer), Windows 10 and 11, ChromeOS, and laptops equipped with DP Alt Mode and Power Delivery. Note: This device is not compatible with Linux.
- What You Get: Anker USB-C Hub (5-in-1, 4K HDMI), welcome guide, 18-month warranty, and our friendly customer service.
import json
from urllib.parse import urlencode
from urllib.request import Request, urlopen
base_url = "https://example.com/api/products"
params = {"category": "books", "page": 1}
url = f"{base_url}?{urlencode(params)}"
request = Request(url, headers={"Accept": "application/json"})
with urlopen(request, timeout=20) as response:
payload = json.load(response)
records = payload.get("items", [])
for record in records:
print(record.get("name"), record.get("price"))For an HTML response, parse its markup with selectors instead of trying to decode it as JSON. For example, install Beautiful Soup with python -m pip install beautifulsoup4, then use BeautifulSoup(html, "html.parser") and selectors matched to the actual response. Keep selectors tolerant of optional fields and verify that the parsed record count is plausible.
Choose the least complex method that gets the data
| Approach | Best fit | Trade-offs |
|---|---|---|
| HTTP client plus HTML or JSON parsing | The fields are in the response or a reproducible data endpoint. | Low browser overhead; you handle pagination, errors, request details, and parsing. |
| Scrapy | You need a reusable crawler or are collecting across multiple pages. | Provides a crawling framework and extraction facilities; dynamic pages may still require identifying the request that supplies the data. |
| Playwright | The result depends on browser rendering, interaction, or a browser-visible state. | Requires browser binaries and more runtime resources; explicit waits help handle timing. Python supports sync and async APIs and Chromium, Firefox, and WebKit. |
| Selenium WebDriver | Browser automation is needed and Selenium fits your existing project or team. | A browser-automation alternative; select it based on project requirements and expertise rather than assuming a universal winner. |
Scrapy’s dynamic-content guidance describes the request-reproduction approach; Selenium’s WebDriver documentation describes its browser automation option. For Playwright, the Python library and installation steps are documented at Getting started.
Use Playwright when the browser is genuinely necessary
Install the Python package and browser binaries as separate steps. The following sync example opens a page, waits for a specific record selector, and extracts text from matching elements. Substitute the target URL, selector, and expected content for the site you are allowed to access.
python -m pip install playwright
python -m playwright installfrom playwright.sync_api import sync_playwright
url = "https://example.com/catalog"
selector = "article.product"
with sync_playwright() as playwright:
browser = playwright.chromium.launch()
page = browser.new_page()
page.goto(url, wait_until="load", timeout=30000)
# Wait for evidence that the records have rendered.
page.locator(selector).first.wait_for(state="visible", timeout=15000)
products = page.locator(selector).all()
for product in products:
print(product.inner_text())
browser.close()The page’s load event is not proof that later data has arrived; a site may fetch it after that event. Playwright documents navigation and loading behavior at Navigations. Prefer waiting for a target locator, a known response, or a site-specific state that demonstrates readiness.
Quick wins for a faster PC:
Repair Windows errors before they cause bigger problemsFix Now →Scan for outdated or missing drivers - takes under a minuteDriver Scan →Clear out junk files and repair common Windows errorsFree Scan →Rank #3
- Sleek 7-in-1 USB-C Hub: Features an HDMI port, two USB-A 3.0 ports, and a USB-C data port, each providing 5Gbps transfer speeds. It also includes a USB-C PD input port for charging up to 100W and dual SD and TF card slots, all in a compact design.
- Flawless 4K@60Hz Video with HDMI: Delivers exceptional clarity and smoothness with its 4K@60Hz HDMI port, making it ideal for high-definition presentations and entertainment. (Note: Only the HDMI port supports video projection; the USB-C port is for data transfer only.)
- Double Up on Efficiency: The two USB-A 3.0 ports and a USB-C port support a fast 5Gbps data rate, significantly boosting your transfer speeds and improving productivity.
- Fast and Reliable 85W Charging: Offers high-capacity, speedy charging for laptops up to 85W, so you spend less time tethered to an outlet and more time being productive.
- What You Get: Anker USB-C Hub (7-in-1), welcome guide, 18-month warranty, and our friendly customer service.
Wait for the condition you actually need
Locators auto-wait for actionability when you perform actions such as clicking. That is different from guaranteeing that an asynchronously changing list has finished loading. In particular, locator.all() returns the matches present at the time it is called; if the list is still changing, the returned collection may not represent the final result. See the Playwright Locator API.
If a site has a reliable “results loaded” element, wait for it before reading the list. If you know the response that carries the records, waiting for that response can be more precise than sleeping for an arbitrary number of seconds. Avoid fixed delays as your only readiness check: they may waste time on fast responses and still be too short on slow ones.
Use the async API for an async application
Playwright also provides an asynchronous Python API. Use it when the surrounding program already uses asyncio; do not mix sync and async calls casually.
import asyncio
from playwright.async_api import async_playwright
async def main():
async with async_playwright() as playwright:
browser = await playwright.chromium.launch()
page = await browser.new_page()
await page.goto("https://example.com/catalog", wait_until="load")
await page.locator("article.product").first.wait_for(state="visible")
names = await page.locator("article.product").all_text_contents()
print(names)
await browser.close()
asyncio.run(main())Or skip the browser setup
If your goal is a rendered page screenshot rather than structured records, ScreenshotNeo can capture a page with one GET request. A screenshot is an image or PDF, not a substitute for extracting structured product or record fields.
Free tools Windows power users keep installed
One-click scans. No signup required.
Rank #4
- Dual Converters, Infinite Potential：Includes 2× USB C male to USB A female adapters and 2× USB A male to USB C female adapters. Perfect for a wide range of uses—tablets with Bluetooth keyboards, expand USB ports on macbook, and more. Two different converters for all your daily needs
- Next-Level 10Gbps & 3A Charging: No more slow 480Mbps, this usb to usb c adapter has a transfer speed of up to 10Gbps, allowing you to do more transferring in less time. This usb adapter fits both USB A and USB C charger, supporting up to 3A fast charging
- Upgraded Exquisite Craftsmanship: With an aluminum alloy housing and metal connector, the usbc to usb adapter is extremely durable and sturdy. Rigorously tested to withstand more than 10,000 times of plugging and unplugging, ensuring long-lasting performance
- Broad Compatible: The usb c to usb adapter widely supports all USB C/ USB A devices like laptops, tablets, cellphones, car chargers, and phone chargers. Such as compatible with MacBook Pro/Air 2023/2022, Thunderbolt 4/3 Devices,Apple MagSafe Watch 9/8/7/SE/Ultra, iPad Pro 2022/2021, Samsung Galaxy S23/S20/S10, and iPhone 17/16/15 Pro. Plug and play
- Please Note: To reach 10Gbps speed, keep the cable under 3.3 ft. For USB A Male to USB C adapters, try flipping the USB C connector. USB C Male to USB A adapters support bidirectional 10Gbps transfer within 3.3 ft
Python example (see the ScreenshotNeo API documentation):
import requests
r = requests.get(
"https://api.screenshotneo.com/v1/shot",
params={"access_key": "YOUR_API_KEY", "url": "https://example.com/catalog"},
timeout=90,
)
open("shot.webp", "wb").write(r.content)ScreenshotNeo removes cookie banners, newsletter popups, and chat widgets before capture; bot checks, blank pages, and failed loads are not billed. Its MCP server lets AI agents take screenshots, and the free plan includes 1,000 screenshots per month with no card; paid plans start at $5 for 3,000. Sign up for 1,000 free screenshots a month, with no card required.
Make collection checkable and conservative
Before scaling a script, verify that it is collecting the intended data and that your access is appropriate. The Robots Exclusion Protocol is standardized in IETF RFC 9309. Python’s urllib.robotparser can parse a robots.txt file and answer whether a user agent may fetch a URL. Robots guidance is not a complete permission or legal analysis; review site-specific terms and applicable law separately.
- Check status codes and content types before parsing.
- Validate record counts and required fields; handle absent values rather than assuming every record is complete.
- For paginated data, identify how the site signals the next page and stop when no further results are available.
- Keep request volume appropriate, and handle timeouts and transient failures without retrying indefinitely.
- Recheck selectors and endpoint behavior when the site changes; neither is guaranteed to remain stable.
Troubleshooting common failures
The response is successful but contains no records
Inspect the response body and content type. The records may come from a later Fetch/XHR request, or the response may be a shell that JavaScript fills in. Find the data request in the Network panel and determine whether it can be reproduced; otherwise use browser automation if rendering is required.
The browser opens, but the selector times out
Confirm the selector against the rendered DOM, not just the initial HTML. Check whether the page requires a different navigation state, a user action, or a more specific wait condition. A timeout can also mean the requested content is unavailable; do not treat every timeout as a reason to increase the wait blindly.
Best Value
- 5-in-1 Connectivity: Equipped with a 4K HDMI port, a 5 Gbps USB-C data port, two 5 Gbps USB-A ports, and a USB C 100W PD-IN port. Note: The USB C 100W PD-IN port supports only charging and does not support data transfer devices such as headphones or speakers.
- Powerful Pass-Through Charging: Supports up to 85W pass-through charging so you can power up your laptop while you use the hub. Note: Pass-through charging requires a charger (not included). Note: To achieve full power for iPad, we recommend using a 45W wall charger.
- Transfer Files in Seconds: Move files to and from your laptop at speeds of up to 5 Gbps via the USB-C and USB-A data ports. Note: The USB C 5Gbps Data port does not support video output.
- HD Display: Connect to the HDMI port to stream or mirror content to an external monitor in resolutions of up to 4K@30Hz. Note: The USB-C ports do not support video output.
- What You Get: Anker 332 USB-C Hub (5-in-1), welcome guide, our worry-free 18-month warranty, and friendly customer service.
The script returns fewer rows than the page shows
The list may be lazy-loaded or still changing when you enumerate it. Determine whether scrolling or pagination is required, wait for evidence that the desired set is ready, and compare the extracted count with the page’s own result count where available. Playwright’s locator.all() reports current matches, not a promise that a changing list is complete.
JSON parsing fails
Check the actual response body and its content type. The endpoint may return HTML, an error page, or a different JSON structure than expected. Print a short body excerpt and status before changing parsing code; then adapt the parser to the observed format.
Requests work manually but fail in automation
Compare the method, URL, query, body, and headers between the working request and your script. Some services depend on cookies or other request state. Only reproduce details needed for a permitted request; do not use this as a way to bypass an access control or challenge.
Recommended Free Tools
Performance, reliability, and maintenance
Direct HTTP requests generally avoid launching a browser, so they are the lighter choice when the response already contains the data. Scrapy is useful when the work is a multi-page crawl with a reusable pipeline. Browser automation has additional installation and execution overhead, but can reproduce browser rendering and interaction when those are essential. These are architectural trade-offs, not guarantees about speed on a particular site.
Keep fetching, parsing, and validation as separate stages. This makes it easier to tell whether a failure comes from a changed endpoint, a changed page structure, or a changed data shape. Use bounded timeouts, surface failures clearly, and avoid treating a partial result as complete. Sites can change their markup, endpoints, policies, and access behavior; build in a way to detect those changes rather than silently accepting empty output.
Frequently Asked Questions
Can I use Playwright and Scrapy in the same Python project?
Yes. They serve different parts of a workflow: Scrapy can manage crawling and extraction, while browser automation can be used where a page genuinely needs rendering or interaction. Choose the integration approach that fits the project rather than adding a browser to every request.
Does robots.txt grant permission to scrape a site?
No. It communicates crawl rules, but it is not a complete permission or legal analysis. Review the site’s terms and applicable requirements separately.
What’s actually slowing this PC down?
Pick the symptom - the matching free tool is one click away.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
