Fetching a web page programmatically means sending an HTTP request, checking the response, and reading its body. For static HTML, a server-side client such as Python’s built-in urllib.request is usually the simplest and most reliable approach. In browser JavaScript, use the promise-based fetch() API when the target permits cross-origin access through CORS. Neither method automatically runs the target page’s JavaScript or reproduces its rendered layout.
The basic fetch sequence
Regardless of language, a production-quality fetch has the same stages:
- Validate the URL. Accept only schemes your application is designed to handle, normally
httpsand, where appropriate,http. - Send an HTTP request. A GET requests the current representation of a resource. GET has no request body and is defined as safe, idempotent, and cacheable.
- Apply a finite timeout. Set connect and read limits so a stalled server cannot hold a worker forever.
- Check the response. Handle status, redirects, authentication challenges, rate limits, and 4xx/5xx responses explicitly.
- Inspect metadata. Read
Content-Typeand the declared character encoding before decoding bytes as HTML or JSON. - Read a bounded body. Cap response bytes to avoid unbounded memory use, then parse or store the result.
Use POST or another method only when an API contract requires a request body or a state change. Query parameters belong in the URL and must be encoded correctly.
Fetch HTML with Python’s standard library
Python 3 includes urllib.request, so no third-party package is required. A Request object lets you set an identifiable User-Agent. If no data argument is supplied, the request uses GET.
#1 Best Overall
- Read Before You Buy — No Video Output: These adapters support charging and USB 2.0 data transfer, but cannot transmit video signals. Except for standard USB webcams (which use USB data only), they are not compatible with HDMI/DisplayPort cables, video-capable USB-C hubs, or docking stations with video output.
- Convert USB-A Ports to USB-C: Designed to connect USB-C earphones, cables, flash drives, card readers, and other USB-C accessories to standard USB-A ports. Plug-and-play with no drivers or software required.
- Aluminum Alloy Housing: Built with a sturdy aluminum alloy shell that aids in heat dissipation and protects against daily wear and scratches. Designed to maintain a stable and secure connection.
- Compact & Travel-Friendly: The ultra-compact design allows the adapter to stay plugged into your device without blocking adjacent ports or adding bulk, reducing wear and tear on your original USB ports.
- 12-Month Warranty: Backed by a 12-month manufacturer warranty for peace of mind. Designed to meet strict quality control standards for reliable everyday performance.
from urllib.request import Request, urlopen
from urllib.error import HTTPError, URLError
url = "https://example.org/"
request = Request(url, headers={"User-Agent": "my-fetcher/1.0"})
try:
with urlopen(request, timeout=10) as response:
status = response.status
content_type = response.headers.get("Content-Type", "")
html = response.read()
if status < 200 or status >= 300:
raise RuntimeError(f"HTTP status {status}")
if "text/html" not in content_type.lower():
raise RuntimeError(f"Unexpected content type: {content_type}")
print(html.decode("utf-8", errors="replace"))
except HTTPError as exc:
print(f"HTTP error: {exc.code}")
except URLError as exc:
print(f"Network/URL error: {exc.reason}")
The context manager closes the response even when processing fails. The official Python documentation demonstrates the same urlopen pattern and documents HTTP/1.1 behavior with a Connection: close header. In a larger crawler, use a client that supports connection reuse when performance requires it.
Decode the response safely
Do not assume every response is UTF-8. First inspect Content-Type for a charset; if it is absent, use a documented fallback appropriate to your application. Keep the original bytes when you need exact archival fidelity. Reject or separately handle compressed, binary, or unexpectedly large responses rather than passing them to an HTML parser blindly.
Fetch a page in browser JavaScript
The browser Fetch API returns a promise for a Response. The promise normally resolves even for HTTP 404 or 504, so test ok or status yourself. Body readers such as text() and json() are asynchronous and can be called only while the body is available.
async function fetchPage(url) {
const response = await fetch(url, { method: "GET" });
if (!response.ok) {
throw new Error(`HTTP ${response.status}`);
}
const contentType = response.headers.get("content-type") || "";
if (!contentType.includes("text/html")) {
throw new Error(`Unexpected content type: ${contentType}`);
}
return await response.text();
}
fetchPage("https://example.org/")
.then(html => console.log(html))
.catch(error => console.error(error));
Abort a slow browser request
async function fetchWithTimeout(url, milliseconds = 10000) {
const controller = new AbortController();
const timer = setTimeout(() => controller.abort(), milliseconds);
try {
const response = await fetch(url, { signal: controller.signal });
if (!response.ok) throw new Error(`HTTP ${response.status}`);
return await response.text();
} finally {
clearTimeout(timer);
}
}
Why browser fetch fails with CORS
Browser JavaScript is constrained by the same-origin policy. A cross-origin Fetch request can be read by your script only when the destination server supplies an appropriate Access-Control-Allow-Origin response header. This is a permission decision made by the destination, not a switch your client can safely override.
Rank #2
- 5-in-1 USB-C Hub: Experience comprehensive connectivity featuring a Power Delivery input, two USB-A 2.0 ports, a USB-A 3.0 port, and an HDMI port. (Note: The USB-C power delivery input port is only for connecting an external wall charger to power your laptop and cannot power peripheral devices.)
- 90W Pass-Through Charging: Achieve optimal charging with 90W pass-through power to your laptop, supported by a total input of 100W, with the hub reserving 10W for operational efficiency. (Note: Wall charger not included.)
- Quick Data Transfers: Accelerate your productivity with rapid data transfers using a high-speed 5Gbps USB 3.0 port and two 480Mbps USB 2.0 ports.
- 4K HDMI Display: Enhance your visual experience with a hub capable of delivering 4K resolution at 30Hz in both mirror and extend modes. Please note that this hub is compatible with MacBook (macOS 12 and newer), Windows 10 and 11, ChromeOS, and laptops equipped with DP Alt Mode and Power Delivery. Note: This device is not compatible with Linux.
- What You Get: Anker USB-C Hub (5-in-1, 4K HDMI), welcome guide, 18-month warranty, and our friendly customer service.
What no-cors does
mode: "no-cors" generally produces an opaque response. Your script cannot read its headers or body, so it is not a solution for downloading another site’s HTML.
Architectures that work
- Fetch from your own server, where cross-origin browser restrictions do not apply to the server-side request.
- Expose a same-origin backend endpoint that validates a supplied URL, performs the request, limits size and time, and returns only data your application is designed to expose.
- Use the destination’s documented cross-origin API when one exists.
A backend proxy must be designed against SSRF: restrict schemes and destinations, block internal address ranges, enforce authentication, cap response size, and log requests. Respect each site’s authentication requirements, robots.txt guidance, rate limits, and terms.
Command-line and Node.js examples
cURL
curl --fail --location --max-time 20
-H "User-Agent: my-fetcher/1.0"
https://example.org/
-o page.html
--fail makes HTTP errors non-successful, --location follows redirects, and --max-time bounds the total operation. Add an explicit byte limit in your surrounding process when downloading untrusted content.
Node.js (built-in Fetch)
const controller = new AbortController();
const timer = setTimeout(() => controller.abort(), 10000);
try {
const response = await fetch('https://example.org/', {
headers: { 'User-Agent': 'my-fetcher/1.0' },
signal: controller.signal
});
if (!response.ok) throw new Error(`HTTP ${response.status}`);
const type = response.headers.get('content-type') || '';
if (!type.includes('text/html')) throw new Error(`Unexpected content type: ${type}`);
const html = await response.text();
console.log(html);
} finally {
clearTimeout(timer);
}
Static HTML versus a JavaScript-rendered application
An HTTP client returns the server’s response body. It does not execute scripts, create browser storage, click controls, wait for client-side requests, or calculate layout. A successful HTTP 200 therefore does not prove that you retrieved the content a visitor sees.
Free tools Windows power users keep installed
One-click scans. No signup required.
Rank #3
- Sleek 7-in-1 USB-C Hub: Features an HDMI port, two USB-A 3.0 ports, and a USB-C data port, each providing 5Gbps transfer speeds. It also includes a USB-C PD input port for charging up to 100W and dual SD and TF card slots, all in a compact design.
- Flawless 4K@60Hz Video with HDMI: Delivers exceptional clarity and smoothness with its 4K@60Hz HDMI port, making it ideal for high-definition presentations and entertainment. (Note: Only the HDMI port supports video projection; the USB-C port is for data transfer only.)
- Double Up on Efficiency: The two USB-A 3.0 ports and a USB-C port support a fast 5Gbps data rate, significantly boosting your transfer speeds and improving productivity.
- Fast and Reliable 85W Charging: Offers high-capacity, speedy charging for laptops up to 85W, so you spend less time tethered to an outlet and more time being productive.
- What You Get: Anker USB-C Hub (7-in-1), welcome guide, 18-month warranty, and our friendly customer service.
Diagnose missing content
- Inspect the fetched HTML for the expected text. If it is present, parse the document normally.
- If the HTML contains an empty app shell, use the site’s documented data endpoint when permitted. Replicate its authentication and request parameters rather than scraping private internals.
- If content requires execution, storage, interaction, or layout, use a permitted browser-automation or rendering service. Set an explicit wait condition, such as a selector, rather than relying only on a fixed delay.
Rendering adds browser startup cost and new failure modes, including bot checks, consent dialogs, resource timeouts, and scripts that behave differently in automation. Use plain HTTP whenever the required data is already in the response.
Reliability, performance, and safe handling
- Timeouts: separate connect and read limits where your client permits it; cancel work that exceeds them.
- Retries: retry only transient network failures and selected 5xx or rate-limit responses, with exponential backoff and a cap. Do not blindly retry authentication or validation errors.
- Connections: reuse connections for repeated requests with a client that supports pooling; Python’s simple example closes each connection.
- Response limits: enforce maximum bytes before buffering. Stream large responses to disk or a parser when appropriate.
- Headers: identify your client truthfully. Do not impersonate a browser to bypass controls.
- Credentials: keep API keys, cookies, and Authorization values out of logs and source control. Send only the credentials required by the site.
- Parsing: classify HTTP, URL, TLS, decoding, and parser errors separately so operators can act on the actual cause.
- Compliance: follow site-specific terms, robots.txt guidance, authentication rules, and rate limits.
Or skip the browser setup
When your real goal is a clean visual capture rather than raw HTML, ScreenshotNeo provides a website screenshot API and MCP server. It accepts a URL in one GET request and returns PNG, JPEG, WebP, or PDF. Before capture it accepts cookie or consent banners like a visitor and removes more than 60 known consent platforms, newsletter popups, and chat widgets; each cleanup step can be disabled. Bot checks, CAPTCHAs, blank pages, timeouts, failed loads, and cache hits are not billed, and response headers identify the page verdict and whether it was billed.
Use the API documentation at https://screenshotneo.com/docs/ for all options, including full-page lazy-image loading, CSS-selector element capture, dark mode, 12 device presets or custom viewports, retina scale, PDF paper size and page ranges, HTML/CSS-to-image, custom JavaScript and CSS, clicks, selector waits, network-idle waits, ad and tracker blocking, custom headers and cookies, user-agent, Authorization, timezone, geolocation, transparent backgrounds, resizing, chosen cache TTL, signed image links, asynchronous webhooks, bulk capture of up to 100 URLs per call, usage data, and the OpenAPI specification.
curl -G "https://api.screenshotneo.com/v1/shot" -d access_key=YOUR_API_KEY --data-urlencode url=https://stripe.com -o shot.webp
import requests
r = requests.get("https://api.screenshotneo.com/v1/shot", params={"access_key": "YOUR_API_KEY", "url": "https://stripe.com"}, timeout=90)
open("shot.webp", "wb").write(r.content)
const q = new URLSearchParams({ access_key: 'YOUR_API_KEY', url: 'https://stripe.com' });
const res = await fetch(`https://api.screenshotneo.com/v1/shot?${q}`);
ScreenshotNeo includes an MCP server with take_screenshot, get_page_info, and capture_pdf tools for Claude, Cursor, and other MCP clients. The Free plan includes 1,000 shots per month without a card; paid plans start at $5 for 3,000 shots. Every feature is available on every plan. Create a free ScreenshotNeo account.
What’s actually slowing this PC down?
Pick the symptom - the matching free tool is one click away.
Troubleshooting common failures
“It returned 404, but fetch did not throw”
Fetch promises reject for network failures, not ordinary HTTP errors. Check response.ok or the numeric status before reading the body.
Rank #4
- Dual Converters, Infinite Potential：Includes 2× USB C male to USB A female adapters and 2× USB A male to USB C female adapters. Perfect for a wide range of uses—tablets with Bluetooth keyboards, expand USB ports on macbook, and more. Two different converters for all your daily needs
- Next-Level 10Gbps & 3A Charging: No more slow 480Mbps, this usb to usb c adapter has a transfer speed of up to 10Gbps, allowing you to do more transferring in less time. This usb adapter fits both USB A and USB C charger, supporting up to 3A fast charging
- Upgraded Exquisite Craftsmanship: With an aluminum alloy housing and metal connector, the usbc to usb adapter is extremely durable and sturdy. Rigorously tested to withstand more than 10,000 times of plugging and unplugging, ensuring long-lasting performance
- Broad Compatible: The usb c to usb adapter widely supports all USB C/ USB A devices like laptops, tablets, cellphones, car chargers, and phone chargers. Such as compatible with MacBook Pro/Air 2023/2022, Thunderbolt 4/3 Devices,Apple MagSafe Watch 9/8/7/SE/Ultra, iPad Pro 2022/2021, Samsung Galaxy S23/S20/S10, and iPhone 17/16/15 Pro. Plug and play
- Please Note: To reach 10Gbps speed, keep the cable under 3.3 ft. For USB A Male to USB C adapters, try flipping the USB C connector. USB C Male to USB A adapters support bidirectional 10Gbps transfer within 3.3 ft
“The browser reports a CORS error”
The destination has not granted your origin read access. Use a permitted same-origin backend, a documented API, or server-side fetching; no-cors will not expose the HTML.
“The response is empty or missing visible text”
The content is likely inserted by JavaScript after the initial response. Locate a documented data endpoint or use permitted rendering with a selector or network-idle wait.
“The request hangs”
Set connect and read or total timeouts, cancel the request, and classify whether the cause is DNS, TLS, a slow server, or an uncompleted response. Retry only transient failures.
Recommended Free Tools
“The parser fails on the result”
Check status and Content-Type, preserve bytes until you know the encoding, and reject binary or oversized responses before parsing as HTML.
Best Value
- 5-in-1 Connectivity: Equipped with a 4K HDMI port, a 5 Gbps USB-C data port, two 5 Gbps USB-A ports, and a USB C 100W PD-IN port. Note: The USB C 100W PD-IN port supports only charging and does not support data transfer devices such as headphones or speakers.
- Powerful Pass-Through Charging: Supports up to 85W pass-through charging so you can power up your laptop while you use the hub. Note: Pass-through charging requires a charger (not included). Note: To achieve full power for iPad, we recommend using a 45W wall charger.
- Transfer Files in Seconds: Move files to and from your laptop at speeds of up to 5 Gbps via the USB-C and USB-A data ports. Note: The USB C 5Gbps Data port does not support video output.
- HD Display: Connect to the HDMI port to stream or mirror content to an external monitor in resolutions of up to 4K@30Hz. Note: The USB-C ports do not support video output.
- What You Get: Anker 332 USB-C Hub (5-in-1), welcome guide, our worry-free 18-month warranty, and friendly customer service.
“A site blocks the fetch”
Do not evade bot controls by spoofing a browser. Confirm that your use is permitted, authenticate as required, reduce request rate, and use an official endpoint or an approved browser workflow.
Choosing the right approach
| Need | Best starting point | Important limitation |
|---|---|---|
| Static HTML from a server | Python urllib.request, Node Fetch, or cURL |
Must handle status, encoding, limits, and timeouts |
| Data from browser code | Browser fetch() |
CORS controls readable cross-origin responses |
| Client-rendered content | Documented data endpoint or permitted browser automation | Plain HTTP does not execute JavaScript |
| Clean visual screenshot or PDF | ScreenshotNeo API or MCP tools | Returns rendered media, not an HTML document |
Frequently Asked Questions
Should I use GET or POST to fetch a page?
Use GET for retrieving a resource. Use POST only when the destination API contract requires a request body or a state-changing operation.
Can I fetch a page directly from frontend JavaScript?
Only when the destination permits your origin through CORS or the request is same-origin. Otherwise route it through a permitted backend or documented API.
The Tool Desk
Outbyte PC Repair FREERepair Windows errors before they cause bigger problemsFix Now →Outbyte Driver Updater FREEFix the driver behind crashes, sound loss and screen glitchesFind Drivers →Does a 200 status mean I received the page users see?
No. It confirms an HTTP response, not that JavaScript-rendered content, browser storage, interactions, or layout were reproduced.
How can I prevent a fetch from consuming unlimited memory?
Set a maximum response size, stream large bodies where supported, inspect Content-Type before parsing, and abort requests that exceed your time limit.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
