If your scraper gets empty HTML from a React, Vue, or Angular site, first check whether the data is in the initial response, embedded in a script, or loaded by a separate request. Use the simplest source that returns the data you need; render the page in a browser only when the content depends on JavaScript execution or browser state.
Why a JavaScript website can return empty HTML
The framework name does not tell you how to scrape a page. React, Vue, and Angular sites may send content in the original HTML, include data in an inline script, or fetch it after the page loads. A server-rendered or pre-rendered page can contain useful content immediately; an app shell may return little more than a page scaffold until JavaScript runs. Google describes this distinction for web apps generally, and notes that not all bots execute JavaScript: Google Search Central’s JavaScript SEO basics.
Also distinguish the original response from the live DOM. A browser’s “View Source” shows the response document, while the Elements panel shows the current DOM after scripts have run and may have changed it.
Diagnose where the target data comes from
- Fetch the page without rendering. Save the response body and search for a distinctive word or value you can see in the browser. Inspect script elements for embedded JSON or other structured data. Compare your scraper’s response with what an ordinary HTTP client receives; this is a recommended first diagnostic in the Scrapy guide to dynamic content.
- Inspect the browser’s network requests. Open Developer Tools, select Network, reload the page, and look for a request whose response contains the missing data. Check Fetch/XHR requests first, but do not assume the data must be JSON or come from an API. It could be in the original document or a JavaScript resource.
- Choose the least complex usable source. If the content is in the original HTML, parse that. If it is embedded as JSON, parse the embedded representation. If a request returns the records in a suitable format, reproduce that request and parse its response. Scrapy recommends locating and reproducing the underlying data request when practical. A discovered endpoint is not automatically stable, authorized, or intended for every use.
- Render only when needed. Use browser automation when the data depends on JavaScript execution, an interaction, or browser-specific state—or when reproducing the data request is impractical. Scrapy documents both request-based extraction and browser-rendering approaches in its dynamic content documentation.
- Wait for an observable condition. Prefer waiting for the target container or a specific result to appear over sleeping for an arbitrary number of seconds. A fixed delay can help diagnose timing, but it does not prove the page is ready. Playwright provides page APIs for browser interaction and waiting: Playwright Page API.
- Validate what you extracted. Check representative fields, record counts, and empty or error states before accepting a run. Client-side route changes, lazy loading, and page updates can change either the request pattern or the selector you rely on.
Choose an extraction method
| What you find | Start with | Reason |
|---|---|---|
| Target content in the raw response HTML | HTTP client and HTML selectors | The response already contains the data, so JavaScript execution is unnecessary. |
| Target content embedded in a script | Parse the embedded representation | It may provide structured data without rendering the page. |
| Target content in a JSON or other data request | Reproduce the request and parse its response | You can often work directly with the data source instead of coordinating a browser. |
| Target content appears only after scripts or browser state | Playwright or another headless browser | A browser can execute page code and expose the resulting DOM. |
| You need crawl orchestration across many pages and browser rendering on some of them | Scrapy with a browser integration | Scrapy’s documentation covers dynamic content and points to browser-integration approaches. |
These choices involve different implementation complexity, completeness, runtime, resource use, and sensitivity to site changes. The sources do not establish a universal speed or cost winner; measure the method against your own pages and data requirements.
Recommended Free Tools
#1 Best Overall
- Read Before You Buy — No Video Output: These adapters support charging and USB 2.0 data transfer, but cannot transmit video signals. Except for standard USB webcams (which use USB data only), they are not compatible with HDMI/DisplayPort cables, video-capable USB-C hubs, or docking stations with video output.
- Convert USB-A Ports to USB-C: Designed to connect USB-C earphones, cables, flash drives, card readers, and other USB-C accessories to standard USB-A ports. Plug-and-play with no drivers or software required.
- Aluminum Alloy Housing: Built with a sturdy aluminum alloy shell that aids in heat dissipation and protects against daily wear and scratches. Designed to maintain a stable and secure connection.
- Compact & Travel-Friendly: The ultra-compact design allows the adapter to stay plugged into your device without blocking adjacent ports or adding bulk, reducing wear and tear on your original USB ports.
- 12-Month Warranty: Backed by a 12-month manufacturer warranty for peace of mind. Designed to meet strict quality control standards for reliable everyday performance.
Extract directly from a data request when practical
In Developer Tools, select the request that returned the desired records and inspect its URL, method, query parameters, headers, request body, and response format. Reproduce only the parts needed to obtain the permitted data. Then parse the response according to its actual format—for example, JSON as JSON rather than as rendered HTML. Scrapy’s guidance is to find where the data comes from and reproduce the relevant request when possible: Scrapy: Dynamic content.
Do not treat a request copied from the browser as a permanent contract. A site can change its endpoint, response shape, or required state. Validate the fields you need and handle missing or changed values rather than silently saving incomplete records.
Rank #2
- 5-in-1 USB-C Hub: Experience comprehensive connectivity featuring a Power Delivery input, two USB-A 2.0 ports, a USB-A 3.0 port, and an HDMI port. (Note: The USB-C power delivery input port is only for connecting an external wall charger to power your laptop and cannot power peripheral devices.)
- 90W Pass-Through Charging: Achieve optimal charging with 90W pass-through power to your laptop, supported by a total input of 100W, with the hub reserving 10W for operational efficiency. (Note: Wall charger not included.)
- Quick Data Transfers: Accelerate your productivity with rapid data transfers using a high-speed 5Gbps USB 3.0 port and two 480Mbps USB 2.0 ports.
- 4K HDMI Display: Enhance your visual experience with a hub capable of delivering 4K resolution at 30Hz in both mirror and extend modes. Please note that this hub is compatible with MacBook (macOS 12 and newer), Windows 10 and 11, ChromeOS, and laptops equipped with DP Alt Mode and Power Delivery. Note: This device is not compatible with Linux.
- What You Get: Anker USB-C Hub (5-in-1, 4K HDMI), welcome guide, 18-month warranty, and our friendly customer service.
Render the page with Playwright when the DOM is the practical source
When the content is only available after the page runs JavaScript, a headless browser can load the page and expose its rendered DOM. This Python example waits for a page-specific selector before reading the result. Install Playwright and its Chromium browser first with pip install playwright and playwright install chromium. Replace the example URL and selector with ones you are permitted to access.
import asyncio
from playwright.async_api import async_playwright
async def main():
async with async_playwright() as p:
browser = await p.chromium.launch()
page = await browser.new_page()
await page.goto("https://example.com", wait_until="domcontentloaded")
# Wait for the specific content you need, not an arbitrary page-wide delay.
await page.locator(".product-list").wait_for(state="visible")
records = await page.locator(".product-card").all_text_contents()
if not records:
raise RuntimeError("The page loaded, but no product records were found")
for record in records:
print(record.strip())
await browser.close()
asyncio.run(main())The selector is illustrative, not a universal convention. If your target content is in a request rather than the DOM, prefer extracting that response where appropriate. For pages whose relevant data arrives later, wait for a selector or another condition tied to the target. Playwright’s documented page APIs are at playwright.dev/docs/api/class-page.
Outdated Drivers Are Slowing You Down
One free scan finds every outdated or missing driver and matches the right update for your exact hardware.Free scan · exact hardware matchPC Slower Than It Used to Be?
A free scan shows the junk files, broken settings and background clutter dragging Windows down - then fixes them in one click.Free scan · Windows 10 & 11Rank #3
- Sleek 7-in-1 USB-C Hub: Features an HDMI port, two USB-A 3.0 ports, and a USB-C data port, each providing 5Gbps transfer speeds. It also includes a USB-C PD input port for charging up to 100W and dual SD and TF card slots, all in a compact design.
- Flawless 4K@60Hz Video with HDMI: Delivers exceptional clarity and smoothness with its 4K@60Hz HDMI port, making it ideal for high-definition presentations and entertainment. (Note: Only the HDMI port supports video projection; the USB-C port is for data transfer only.)
- Double Up on Efficiency: The two USB-A 3.0 ports and a USB-C port support a fast 5Gbps data rate, significantly boosting your transfer speeds and improving productivity.
- Fast and Reliable 85W Charging: Offers high-capacity, speedy charging for laptops up to 85W, so you spend less time tethered to an outlet and more time being productive.
- What You Get: Anker USB-C Hub (7-in-1), welcome guide, 18-month warranty, and our friendly customer service.
Handle crawl boundaries and access responsibly
Check the site’s terms, access controls, and applicable legal requirements before collecting data, particularly where authentication, personal information, copyrighted material, or other restrictions are involved. Legal outcomes depend on jurisdiction and facts.
RFC 9309, the IETF Robots Exclusion Protocol standard published in September 2022, describes robots.txt as crawler guidance, not permission to access restricted material. Its text states: “These rules are not a form of access authorization.” Read the standard at RFC 9309. An allowed path in robots.txt does not grant access to protected content.
Rank #4
- Dual Converters, Infinite Potential：Includes 2× USB C male to USB A female adapters and 2× USB A male to USB C female adapters. Perfect for a wide range of uses—tablets with Bluetooth keyboards, expand USB ports on macbook, and more. Two different converters for all your daily needs
- Next-Level 10Gbps & 3A Charging: No more slow 480Mbps, this usb to usb c adapter has a transfer speed of up to 10Gbps, allowing you to do more transferring in less time. This usb adapter fits both USB A and USB C charger, supporting up to 3A fast charging
- Upgraded Exquisite Craftsmanship: With an aluminum alloy housing and metal connector, the usbc to usb adapter is extremely durable and sturdy. Rigorously tested to withstand more than 10,000 times of plugging and unplugging, ensuring long-lasting performance
- Broad Compatible: The usb c to usb adapter widely supports all USB C/ USB A devices like laptops, tablets, cellphones, car chargers, and phone chargers. Such as compatible with MacBook Pro/Air 2023/2022, Thunderbolt 4/3 Devices,Apple MagSafe Watch 9/8/7/SE/Ultra, iPad Pro 2022/2021, Samsung Galaxy S23/S20/S10, and iPhone 17/16/15 Pro. Plug and play
- Please Note: To reach 10Gbps speed, keep the cable under 3.3 ft. For USB A Male to USB C adapters, try flipping the USB C connector. USB C Male to USB A adapters support bidirectional 10Gbps transfer within 3.3 ft
Keep crawl behavior within the access conditions that apply to your use. This guide does not recommend bypassing access controls or defeating anti-bot measures.
Independent reader supportYour contribution helps us test, update, and keep practical guides available for everyone.Troubleshoot common failures
- The HTTP response is empty of the visible content. Inspect the page’s network activity for the request that returns the data. If the content is embedded in a script, inspect that representation; if it appears only after execution, use browser rendering.
- The browser loads, but the selector times out. Confirm the selector against the live DOM after the page has loaded. Check whether the page navigated to a different route, whether the content is inside a frame, or whether the target appears only after an interaction. Wait for a condition tied to the actual content.
- The selector succeeds but returns no records. Check for an empty-state message, an error state, or a result list that has not finished updating. Verify that the selector matches the intended repeated elements and that the page has the expected data.
- Your direct request no longer works. The page may have changed its request pattern or response shape, or the request may depend on state. Reinspect the current network request and validate the response before parsing it.
- Results are incomplete. Check whether the site loads more content as you scroll, changes routes client-side, or updates its results after interaction. Determine the site’s actual loading behavior rather than assuming one initial response contains every record.
Or skip the browser setup
ScreenshotNeo is a website screenshot API and MCP server. For a visual capture of a JavaScript-rendered page, one GET request returns an image or PDF; it produces screenshots, not extracted page records. See the ScreenshotNeo API documentation for options and response details.
The Tool Desk
Outbyte Driver Updater FREEScan for outdated or missing drivers - takes under a minuteDriver Scan →Outbyte PC Repair FREEClear out junk files and repair common Windows errorsFree Scan →Best Value
- 5-in-1 Connectivity: Equipped with a 4K HDMI port, a 5 Gbps USB-C data port, two 5 Gbps USB-A ports, and a USB C 100W PD-IN port. Note: The USB C 100W PD-IN port supports only charging and does not support data transfer devices such as headphones or speakers.
- Powerful Pass-Through Charging: Supports up to 85W pass-through charging so you can power up your laptop while you use the hub. Note: Pass-through charging requires a charger (not included). Note: To achieve full power for iPad, we recommend using a 45W wall charger.
- Transfer Files in Seconds: Move files to and from your laptop at speeds of up to 5 Gbps via the USB-C and USB-A data ports. Note: The USB C 5Gbps Data port does not support video output.
- HD Display: Connect to the HDMI port to stream or mirror content to an external monitor in resolutions of up to 4K@30Hz. Note: The USB-C ports do not support video output.
- What You Get: Anker 332 USB-C Hub (5-in-1), welcome guide, our worry-free 18-month warranty, and friendly customer service.
curl -G "https://api.screenshotneo.com/v1/shot" -d access_key=YOUR_API_KEY --data-urlencode url=https://example.com -o shot.webpBefore capture, ScreenshotNeo can accept cookie or consent banners and remove more than 60 known consent platforms, newsletter popups, and chat widgets; each step can be turned off. Bot checks or CAPTCHAs, blank pages, timeouts, failed loads, and cache hits are not billed, and response headers report the page verdict and billing status. Its MCP server provides take_screenshot, get_page_info, and capture_pdf for AI agents and MCP clients. The free plan includes 1,000 screenshots per month with no card required; paid plans start at $5 for 3,000 screenshots.
Sign up for 1,000 free screenshots a month, with no card.
Further reading
For a broader Python scraping reference, O’Reilly lists Ryan Mitchell’s Web Scraping with Python, 3rd Edition as published in February 2024, with 352 pages and coverage of scraping JavaScript and crawling through APIs. It is listed for intermediate to advanced readers; it is optional, not a requirement for the workflow above. O’Reilly book listing.
Frequently Asked Questions
Does React, Vue, or Angular require a different scraping method?
No. The relevant question is where the page’s data is delivered: in the initial response, embedded script data, a separate request, or only after browser execution.
Quick wins for a faster PC:
Fix the driver behind crashes, sound loss and screen glitchesFind Drivers →Clear out junk files and repair common Windows errorsFree Scan →Scan for outdated or missing drivers - takes under a minuteDriver Scan →Is a robots.txt allow rule permission to scrape a page?
No. RFC 9309 says robots.txt rules are not access authorization; check applicable access conditions separately.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
