To scrape a JavaScript-rendered website with Python, first check whether a normal HTTP response already contains the data. If it does, parse that response with an HTTP client such as Requests. If the data appears only after the page runs JavaScript, use browser automation—such as Playwright or Selenium—then wait for the specific content or network response you need before extracting it.
First check whether you need a browser
A page can look like it contains data in a browser while its initial HTML response does not. The site may build the content in the browser after JavaScript runs, or fetch it later from an API. A plain HTTP client does not execute that page JavaScript; Requests is an HTTP library for sending requests and handling responses (Requests documentation).
- Fetch the page with Requests and inspect the response text for the data or relevant structured markup.
- If the data is present, parse the response directly. This is usually the simpler approach.
- If the data is missing because client-side scripts create it, use a browser automation library.
Do not infer that a page is JavaScript-rendered just because it uses JavaScript. The practical question is whether the response you can retrieve directly contains the specific information you need.
Choose Playwright or Selenium
Both Playwright and Selenium are reasonable options for browser-driven scraping. Neither is universally best; choose based on the APIs, browser environment, and project setup that suit your task.
#1 Best Overall
- Read Before You Buy — No Video Output: These adapters support charging and USB 2.0 data transfer, but cannot transmit video signals. Except for standard USB webcams (which use USB data only), they are not compatible with HDMI/DisplayPort cables, video-capable USB-C hubs, or docking stations with video output.
- Convert USB-A Ports to USB-C: Designed to connect USB-C earphones, cables, flash drives, card readers, and other USB-C accessories to standard USB-A ports. Plug-and-play with no drivers or software required.
- Aluminum Alloy Housing: Built with a sturdy aluminum alloy shell that aids in heat dissipation and protects against daily wear and scratches. Designed to maintain a stable and secure connection.
- Compact & Travel-Friendly: The ultra-compact design allows the adapter to stay plugged into your device without blocking adjacent ports or adding bulk, reducing wear and tear on your original USB ports.
- 12-Month Warranty: Backed by a 12-month manufacturer warranty for peace of mind. Designed to meet strict quality control standards for reliable everyday performance.
| Consideration | Playwright | Selenium |
|---|---|---|
| Page interaction and extraction | Provides locators, page-context JavaScript evaluation, and network monitoring (locators, evaluation, network). | Automates browsers through its WebDriver model; see the Python API documentation. |
| Waiting | Can wait for a locator, navigation condition, or network response. Its navigation guide explains why the load event alone may not mean the app’s data is ready (navigation guide). | Documents explicit and implicit wait strategies (waiting strategies). |
| Environment and remote execution | Check the current official documentation for the browser and runtime setup supported by your chosen version. | The Python API documentation lists supported browser integrations and a Remote protocol. It also describes Selenium Manager handling driver and browser setup on most supported platforms. |
| Existing project fit | Often a natural choice when its locator, evaluation, and response-wait APIs fit the task. | Often a natural choice when a project already uses WebDriver or its browser and remote-execution model. |
The Selenium Python API documentation currently states Python 3.10 or later support and displayed version 4.50.0 when consulted. These details can change; confirm the requirements for the version you install.
Scrape rendered content with Playwright
Install Playwright for Python and its browser binaries in the environment where the script will run. Use the current Playwright Python installation instructions for the exact commands and browser setup for your platform. The following synchronous example illustrates the workflow; the selector is site-specific and should be replaced with one that identifies the target content.
from playwright.sync_api import sync_playwright
with sync_playwright() as p:
browser = p.chromium.launch()
page = browser.new_page()
page.goto("https://example.com")
page.locator("main article").wait_for()
rendered_text = page.locator("main article").inner_text()
print(rendered_text)
browser.close()page.goto() handles navigation, but its completion is not proof that a modern application has finished loading the data you want. Playwright’s navigation documentation notes: “Modern pages perform numerous activities after the ‘load’ event was fired. They fetch data lazily, populate UI, load expensive resources, scripts and styles after the ‘load’ event was fired.” Wait for a locator, text, URL, or response that demonstrates the state your scraper needs (Playwright navigation guide).
Pass values into page JavaScript explicitly
When extraction is easier with a DOM expression, use page.evaluate(). Its code runs in the browser’s page environment, separate from Python. Pass Python values as explicit arguments rather than assuming Python variables are available inside the page.
Free tools Windows power users keep installed
One-click scans. No signup required.
Rank #2
- 5-in-1 USB-C Hub: Experience comprehensive connectivity featuring a Power Delivery input, two USB-A 2.0 ports, a USB-A 3.0 port, and an HDMI port. (Note: The USB-C power delivery input port is only for connecting an external wall charger to power your laptop and cannot power peripheral devices.)
- 90W Pass-Through Charging: Achieve optimal charging with 90W pass-through power to your laptop, supported by a total input of 100W, with the hub reserving 10W for operational efficiency. (Note: Wall charger not included.)
- Quick Data Transfers: Accelerate your productivity with rapid data transfers using a high-speed 5Gbps USB 3.0 port and two 480Mbps USB 2.0 ports.
- 4K HDMI Display: Enhance your visual experience with a hub capable of delivering 4K resolution at 30Hz in both mirror and extend modes. Please note that this hub is compatible with MacBook (macOS 12 and newer), Windows 10 and 11, ChromeOS, and laptops equipped with DP Alt Mode and Power Delivery. Note: This device is not compatible with Linux.
- What You Get: Anker USB-C Hub (5-in-1, 4K HDMI), welcome guide, 18-month warranty, and our friendly customer service.
from playwright.sync_api import sync_playwright
with sync_playwright() as p:
browser = p.chromium.launch()
page = browser.new_page()
page.goto("https://example.com")
page.locator("main article").wait_for()
selector = "main article"
text = page.evaluate(
"selector => document.querySelector(selector)?.innerText ?? null",
selector,
)
print(text)
browser.close()See Playwright’s evaluation documentation for the page/Python boundary and supported evaluation patterns.
Wait for an in-place update
If clicking a button updates the page without navigating, wait for the resulting content or the response that supplies it. A click does not necessarily trigger navigation, so a navigation wait is not the right condition for every interaction.
from playwright.sync_api import sync_playwright
with sync_playwright() as p:
browser = p.chromium.launch()
page = browser.new_page()
page.goto("https://example.com")
page.get_by_role("button", name="Load results").click()
page.locator("[data-testid='results'] li").first.wait_for()
results = page.locator("[data-testid='results'] li").all_inner_texts()
print(results)
browser.close()Replace the button and result selectors with those on the target site. Playwright locators support element lookup and interaction (locator documentation).
Use Selenium when WebDriver fits your setup
Selenium automates browser interaction through WebDriver. The example below uses Selenium’s Python API and an explicit wait for the content rather than assuming a fixed delay is enough.
Rank #3
- Sleek 7-in-1 USB-C Hub: Features an HDMI port, two USB-A 3.0 ports, and a USB-C data port, each providing 5Gbps transfer speeds. It also includes a USB-C PD input port for charging up to 100W and dual SD and TF card slots, all in a compact design.
- Flawless 4K@60Hz Video with HDMI: Delivers exceptional clarity and smoothness with its 4K@60Hz HDMI port, making it ideal for high-definition presentations and entertainment. (Note: Only the HDMI port supports video projection; the USB-C port is for data transfer only.)
- Double Up on Efficiency: The two USB-A 3.0 ports and a USB-C port support a fast 5Gbps data rate, significantly boosting your transfer speeds and improving productivity.
- Fast and Reliable 85W Charging: Offers high-capacity, speedy charging for laptops up to 85W, so you spend less time tethered to an outlet and more time being productive.
- What You Get: Anker USB-C Hub (7-in-1), welcome guide, 18-month warranty, and our friendly customer service.
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.support.ui import WebDriverWait
# Selenium Manager handles driver setup on most supported platforms
# with modern Selenium versions.
driver = webdriver.Chrome()
try:
driver.get("https://example.com")
article = WebDriverWait(driver, 15).until(
EC.visibility_of_element_located((By.CSS_SELECTOR, "main article"))
)
print(article.text)
finally:
driver.quit()Install Selenium and confirm the browser and runtime requirements for the version in your environment using the Selenium Python API documentation. Selenium documents explicit and implicit waits; an explicit wait is useful when a particular element or state is the readiness condition (Selenium waiting strategies).
Wait for the evidence that your data is ready
Choose a wait condition that matches what the page does:
- Navigation: wait for the expected URL or destination content.
- Content added to the current page: wait for the target locator or text to appear.
- Button or form triggers data: wait for the relevant network response or for the returned data to show in the page.
- Lazy-loaded content: trigger the required scroll or interaction, then wait for the newly loaded items.
Fixed sleeps can be useful for diagnosis, but they are a poor primary synchronization method: they may waste time when a page is ready quickly and still be too short when it is slow. Use the condition that actually proves readiness. Playwright describes navigation and post-load activity in its navigation guide; Selenium covers its wait strategies in the waiting documentation.
Inspect network traffic for the data source
If the DOM route is unclear, inspect the browser’s network traffic while loading the page or performing the interaction that reveals the data. Look for the XHR or fetch request whose response contains the records. Playwright can monitor HTTP and HTTPS traffic and wait for a response associated with an action (network documentation).
Do these 3 things before closing this tab:
1Repair Windows errors before they cause bigger problems2Fix the driver behind crashes, sound loss and screen glitches3Clear out junk files and repair common Windows errorsRank #4
- Dual Converters, Infinite Potential：Includes 2× USB C male to USB A female adapters and 2× USB A male to USB C female adapters. Perfect for a wide range of uses—tablets with Bluetooth keyboards, expand USB ports on macbook, and more. Two different converters for all your daily needs
- Next-Level 10Gbps & 3A Charging: No more slow 480Mbps, this usb to usb c adapter has a transfer speed of up to 10Gbps, allowing you to do more transferring in less time. This usb adapter fits both USB A and USB C charger, supporting up to 3A fast charging
- Upgraded Exquisite Craftsmanship: With an aluminum alloy housing and metal connector, the usbc to usb adapter is extremely durable and sturdy. Rigorously tested to withstand more than 10,000 times of plugging and unplugging, ensuring long-lasting performance
- Broad Compatible: The usb c to usb adapter widely supports all USB C/ USB A devices like laptops, tablets, cellphones, car chargers, and phone chargers. Such as compatible with MacBook Pro/Air 2023/2022, Thunderbolt 4/3 Devices,Apple MagSafe Watch 9/8/7/SE/Ultra, iPad Pro 2022/2021, Samsung Galaxy S23/S20/S10, and iPhone 17/16/15 Pro. Plug and play
- Please Note: To reach 10Gbps speed, keep the cable under 3.3 ft. For USB A Male to USB C adapters, try flipping the USB C connector. USB C Male to USB A adapters support bidirectional 10Gbps transfer within 3.3 ft
from playwright.sync_api import sync_playwright
with sync_playwright() as p:
browser = p.chromium.launch()
page = browser.new_page()
page.goto("https://example.com")
with page.expect_response(
lambda response: "/api/results" in response.url
and response.request.method == "GET"
) as response_info:
page.get_by_role("button", name="Load results").click()
response = response_info.value
print(response.status)
print(response.text())
browser.close()Change the URL condition, HTTP method, and triggering action to match the site. If the endpoint is accessible and the site’s rules permit its use, requesting it directly may avoid repeatedly rendering the full page. Check its parameters, pagination, and response shape: one response may not contain all results. Discovering a request is not proof that you are permitted to automate access to it.
Extract and validate the results
Once the data is ready, extract it from a stable locator or an evaluated DOM expression. Then check that the output is actually complete enough for your task:
- Confirm the result is non-empty and the expected fields are present.
- Check record counts against what the page indicates, when available.
- Verify the page, query, filters, or account context matches the intended scrape.
- Handle pagination or scrolling when the site loads results in batches.
- Record failures explicitly rather than treating a completed navigation as a successful extraction.
Selectors must match the actual site. The examples here show patterns; they are not a verified scrape of a particular live target.
Independent reader supportYour contribution helps us test, update, and keep practical guides available for everyone.Troubleshoot common failures
| Symptom | Likely cause | What to try |
|---|---|---|
| Requests returns no target data | The response contains a shell or placeholder and the browser creates the content later. | Inspect the response and network traffic. Use browser automation if scripts render the data, or use an accessible data endpoint if the site’s rules permit it. |
| Navigation completes but the selector times out | The content is lazy-loaded, delayed, behind an interaction, or the selector does not match. | Inspect the live DOM, verify the selector, and wait for the actual update condition. Trigger required scrolling or interaction. |
| A click appears to do nothing | The page may not have attached its event listener yet, the locator may select the wrong control, or the control is blocked by an overlay. | Wait for the control to be actionable, confirm its accessible name or selector, and inspect whether the action causes a response or navigation. |
| The scraper reads old or incomplete results | The application updated asynchronously or paginated results were not fetched. | Wait for the expected response or changed content; verify pagination, query parameters, and record counts. |
| The direct endpoint request fails or returns a different shape | The request may need parameters, headers, cookies, or a different method, or the endpoint may not be available to your client. | Compare the observed request and response with the browser’s network panel. Do not assume an endpoint is public or stable. |
| Browser or driver setup fails | The installed package, browser, driver, or runtime requirements may not match. | Follow the current official installation steps and verify version compatibility. Selenium Manager handles driver and browser setup on most supported platforms for modern Selenium versions; it is not a guarantee that every environment is configured correctly. |
Reliability, performance, and responsible access
Direct HTTP requests generally avoid the work of starting and controlling a browser when the needed data is already in a response. Browser automation adds rendering and interaction steps, so use it only when the content or workflow requires a browser. No comparative speed benchmark is established here; performance depends on the target page, waits, browser environment, and volume.
Best Value
- 5-in-1 Connectivity: Equipped with a 4K HDMI port, a 5 Gbps USB-C data port, two 5 Gbps USB-A ports, and a USB C 100W PD-IN port. Note: The USB C 100W PD-IN port supports only charging and does not support data transfer devices such as headphones or speakers.
- Powerful Pass-Through Charging: Supports up to 85W pass-through charging so you can power up your laptop while you use the hub. Note: Pass-through charging requires a charger (not included). Note: To achieve full power for iPad, we recommend using a 45W wall charger.
- Transfer Files in Seconds: Move files to and from your laptop at speeds of up to 5 Gbps via the USB-C and USB-A data ports. Note: The USB C 5Gbps Data port does not support video output.
- HD Display: Connect to the HDMI port to stream or mirror content to an external monitor in resolutions of up to 4K@30Hz. Note: The USB-C ports do not support video output.
- What You Get: Anker 332 USB-C Hub (5-in-1), welcome guide, our worry-free 18-month warranty, and friendly customer service.
For repeated collection, make readiness checks specific, handle pagination deliberately, and distinguish missing data from timeouts or navigation errors in your output. Selenium’s Remote protocol can be relevant when execution needs to be separated from the local script, but the appropriate deployment depends on your environment (Selenium Python API documentation).
Technical ability to retrieve a page or endpoint does not establish permission to collect its data. Check applicable site terms, access controls, and relevant law; do not treat browser automation as a way to defeat bot checks or restrictions.
Or skip the browser setup
If your goal is a screenshot rather than structured records, ScreenshotNeo is a website screenshot API and MCP server for developers. Its GET endpoint returns a PNG, JPEG, WebP, or PDF for a URL. See the API documentation for options.
curl -G "https://api.screenshotneo.com/v1/shot" -d access_key=YOUR_API_KEY --data-urlencode url=https://example.com -o shot.webpScreenshotNeo accepts cookie or consent banners like a visitor and removes more than 60 known consent platforms, newsletter popups, and chat widgets before capture; each step can be turned off. Bot checks or CAPTCHAs, blank pages, timeouts, failed loads, and cache hits are not billed, and responses identify the page verdict and billing status in headers. Its MCP server provides take_screenshot, get_page_info, and capture_pdf tools for AI agents. The free plan includes 1,000 screenshots per month with no card; paid plans start at $5 for 3,000 screenshots. These are ScreenshotNeo plan terms, not a substitute for checking whether you need DOM data rather than an image.
Sign up for 1,000 free screenshots a month with no card.
Frequently Asked Questions
Can Requests scrape a JavaScript-rendered website by itself?
Requests retrieves HTTP responses but does not run the page’s JavaScript. If the needed data is absent from the response, use browser automation or investigate an accessible data request.
Should I use Playwright or Selenium for Python scraping?
Either can work. Choose based on your browser and execution environment, interaction and waiting needs, and the library your project already supports.
Is it legal to scrape a website with Python?
That depends on the site, data, access conditions, and applicable law. Review the site’s terms and access controls and seek appropriate legal advice when needed.
The Tool Desk
Outbyte PC Repair FREEClear out junk files and repair common Windows errorsFree Scan →Outbyte Driver Updater FREEScan for outdated or missing drivers - takes under a minuteDriver Scan →Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
