There is no universal set of HTTP headers that makes a web scraper legitimate or guarantees that a site will let it through. Headers describe parts of a request—such as the client, desired format, language, or session—but they do not override a site’s access rules. For authorized scraping, check the site’s policy first, send only headers the documented workflow requires, and diagnose the response before changing anything.
What headers can—and cannot—do
An HTTP header supplies information or instructions associated with a request. A server may use it to choose a representation, identify a session, apply a cache rule, or evaluate a request. The exact behavior depends on the site, any proxy or CDN between you and the origin, and the client runtime.
Headers cannot establish permission on their own. A site may require authentication, provide a supported API, or deny automated requests through server-side controls. Changing a browser-looking header is not a reliable or appropriate way to defeat those controls.
- Useful: sending an accurate
Acceptvalue, using the site’s documented authentication flow, or identifying an authorized crawler honestly. - Not a guarantee: copying a browser’s
User-Agentor inventingReferer,Origin, orSec-Fetch-*values. - Not authorization: a permissive
robots.txtentry, a successful response to one request, or a header that resembles another client.
Check permission and the intended access path first
Before writing a scraper, look for a documented API, export, feed, or crawl policy. Confirm that your planned use and request volume are permitted. If the site denies access, stop and ask for authorization or use a supported alternative rather than cycling through spoofed headers.
Do these 3 things before closing this tab:
1Clear out junk files and repair common Windows errors2Scan for outdated or missing drivers - takes under a minute3Repair Windows errors before they cause bigger problems#1 Best Overall
- Read Before You Buy — No Video Output: These adapters support charging and USB 2.0 data transfer, but cannot transmit video signals. Except for standard USB webcams (which use USB data only), they are not compatible with HDMI/DisplayPort cables, video-capable USB-C hubs, or docking stations with video output.
- Convert USB-A Ports to USB-C: Designed to connect USB-C earphones, cables, flash drives, card readers, and other USB-C accessories to standard USB-A ports. Plug-and-play with no drivers or software required.
- Aluminum Alloy Housing: Built with a sturdy aluminum alloy shell that aids in heat dissipation and protects against daily wear and scratches. Designed to maintain a stable and secure connection.
- Compact & Travel-Friendly: The ultra-compact design allows the adapter to stay plugged into your device without blocking adjacent ports or adding bulk, reducing wear and tear on your original USB ports.
- 12-Month Warranty: Backed by a 12-month manufacturer warranty for peace of mind. Designed to meet strict quality control standards for reliable everyday performance.
robots.txt is a voluntary crawler convention, not an access-control mechanism. Cloudflare’s guidance says well-behaved bots follow it, but a server cannot rely on the file alone to enforce restrictions. Site owners that need enforcement should use server-side controls such as authentication, request validation, or WAF rules. Treat any published crawl delay as a policy to honor; support for directives such as Crawl-delay varies by crawler. Cloudflare’s robots.txt guidance gives a two-second crawl-delay example and recommends listing sitemap locations for discovery.
Use headers to describe the real request
User-Agent: identify your client honestly
The User-Agent is a declared client string, not proof of identity. If the target asks crawlers to identify themselves, use a truthful value and include a contact or project page when appropriate. Do not copy a current desktop browser string as a magic bypass.
This distinction is explicit in Cloudflare’s Browser Run documentation: “The User-Agent header is not a reliable way to identify Browser Run requests.” That statement is specifically about identifying Browser Run requests. Cloudflare says the value is configurable for most Browser Run methods, can change with the underlying Chrome version, and can be sent by any HTTP client. Its documentation describes non-configurable headers and Web Bot Auth signatures as a stronger identification path for that service; this is not a universal description of every site’s detection system. Cloudflare Browser Run automatic request headers (last updated June 16, 2026).
Accept and Accept-Language: request representations you can use
Set Accept to formats your client can process, and Accept-Language to a language preference that reflects the application or user. These headers can influence content selection and cache behavior. They are not established as reliable ways to prevent blocks. Cloudflare Workers documentation describes normalizing these values for cache variation; that is cache correctness guidance, not anti-bot advice.
What’s actually slowing this PC down?
Pick the symptom - the matching free tool is one click away.
Rank #2
- 5-in-1 USB-C Hub: Experience comprehensive connectivity featuring a Power Delivery input, two USB-A 2.0 ports, a USB-A 3.0 port, and an HDMI port. (Note: The USB-C power delivery input port is only for connecting an external wall charger to power your laptop and cannot power peripheral devices.)
- 90W Pass-Through Charging: Achieve optimal charging with 90W pass-through power to your laptop, supported by a total input of 100W, with the hub reserving 10W for operational efficiency. (Note: Wall charger not included.)
- Quick Data Transfers: Accelerate your productivity with rapid data transfers using a high-speed 5Gbps USB 3.0 port and two 480Mbps USB 2.0 ports.
- 4K HDMI Display: Enhance your visual experience with a hub capable of delivering 4K resolution at 30Hz in both mirror and extend modes. Please note that this hub is compatible with MacBook (macOS 12 and newer), Windows 10 and 11, ChromeOS, and laptops equipped with DP Alt Mode and Power Delivery. Note: This device is not compatible with Linux.
- What You Get: Anker USB-C Hub (5-in-1, 4K HDMI), welcome guide, 18-month warranty, and our friendly customer service.
Accept-Encoding: let the library manage compression
HTTP libraries commonly negotiate compression and decompress responses for you. Prefer their built-in behavior rather than manually advertising encodings your code cannot decode. The network path can change what the origin sees: Cloudflare documents that for requests passing through its network, it sets the incoming Accept-Encoding value to br, gzip before forwarding to the origin. That is a Cloudflare-specific transformation, not a rule for all servers. Cloudflare HTTP headers reference.
Cookie: use a session flow, not copied secrets
When an authorized workflow requires state, use the client library’s cookie jar or the application’s documented login/session mechanism. Avoid hardcoding session cookies, publishing them in source control, or reusing them across unrelated users or jobs. Cookies are credentials.
Runtime matters. Browser JavaScript cannot directly set the Cookie request header; the browser manages cookies. Cloudflare Workers treat Cookie as an ordinary header. A server-side Python or Node client may let your code set it, but that does not make a copied cookie safe or authorized.
Referer, Origin, and browser-generated headers
Send these only when the real application flow requires them and their values are truthful. The official material cited here does not establish a universal set of Referer, Origin, or Sec-Fetch-* values that unlock access. Fabricating them to imitate a browser is not a sound diagnostic method.
Outdated Drivers Are Slowing You Down
One free scan finds every outdated or missing driver and matches the right update for your exact hardware.Free scan · exact hardware matchPC Slower Than It Used to Be?
A free scan shows the junk files, broken settings and background clutter dragging Windows down - then fixes them in one click.Free scan · Windows 10 & 11Rank #3
- Sleek 7-in-1 USB-C Hub: Features an HDMI port, two USB-A 3.0 ports, and a USB-C data port, each providing 5Gbps transfer speeds. It also includes a USB-C PD input port for charging up to 100W and dual SD and TF card slots, all in a compact design.
- Flawless 4K@60Hz Video with HDMI: Delivers exceptional clarity and smoothness with its 4K@60Hz HDMI port, making it ideal for high-definition presentations and entertainment. (Note: Only the HDMI port supports video projection; the USB-C port is for data transfer only.)
- Double Up on Efficiency: The two USB-A 3.0 ports and a USB-C port support a fast 5Gbps data rate, significantly boosting your transfer speeds and improving productivity.
- Fast and Reliable 85W Charging: Offers high-capacity, speedy charging for laptops up to 85W, so you spend less time tethered to an outlet and more time being productive.
- What You Get: Anker USB-C Hub (7-in-1), welcome guide, 18-month warranty, and our friendly customer service.
Provider and proxy headers belong to that network path
Do not invent CF-*, X-Forwarded-*, or client-IP headers to impersonate a proxy or network. Cloudflare documents adding or transforming headers between its edge and an origin behind Cloudflare, including CF-Connecting-IP. Their meanings depend on that provider architecture; sending a similarly named header from an unrelated client does not reproduce it.
Diagnose a block without guessing
- Confirm permission and supported access. Check the site’s terms or published crawl policy and look for an API, feed, or export.
- Reproduce the authorized request. Keep the URL, method, authentication state, and representation needs consistent with the documented browser or API workflow.
- Inspect the response before editing headers. Record the status code, redirect chain, content type, and response body. A 403, challenge page, login redirect, or empty body points to different issues.
- Check your client’s behavior. Verify redirect handling, cookie-jar configuration, compression/decompression, and whether the runtime permits setting the header in question.
- Add only documented requirements. Keep the client identity truthful, use the proper session flow, and avoid stale cookies or copied secrets.
- Respect a continuing denial. If the site still refuses the request, seek permission or switch to an authorized route; do not escalate through spoofed values.
A 403 is not a diagnosis by itself. It says the server refused the request; the body, response headers, redirect behavior, and site documentation may help explain why, but the material here does not establish the internal rules of non-Cloudflare anti-bot systems.
Make redirects and credentials safe
Redirects can change where credentials go. Cloudflare warns that a Worker fetch() configured to follow redirects can forward sensitive headers, including Cookie and Authorization, to the redirect destination, even across hostnames. If your code forwards credentials, choose a redirect policy deliberately: inspect the destination and do not forward secrets to an untrusted host. Review the runtime’s rules instead of assuming Python, Node.js, browser JavaScript, and Workers behave identically. Cloudflare Workers Request documentation.
Respect crawl scope and pace
For a compliant crawler, keep the scope narrow, use published sitemap URLs where appropriate, and apply the site’s stated pacing policy. A delay is not a permission grant, and robots.txt does not enforce itself; it is guidance for crawlers to honor.
The Tool Desk
Outbyte PC Repair FREERepair Windows errors before they cause bigger problemsFix Now →Outbyte Driver Updater FREEScan for outdated or missing drivers - takes under a minuteDriver Scan →Rank #4
- Dual Converters, Infinite Potential：Includes 2× USB C male to USB A female adapters and 2× USB A male to USB C female adapters. Perfect for a wide range of uses—tablets with Bluetooth keyboards, expand USB ports on macbook, and more. Two different converters for all your daily needs
- Next-Level 10Gbps & 3A Charging: No more slow 480Mbps, this usb to usb c adapter has a transfer speed of up to 10Gbps, allowing you to do more transferring in less time. This usb adapter fits both USB A and USB C charger, supporting up to 3A fast charging
- Upgraded Exquisite Craftsmanship: With an aluminum alloy housing and metal connector, the usbc to usb adapter is extremely durable and sturdy. Rigorously tested to withstand more than 10,000 times of plugging and unplugging, ensuring long-lasting performance
- Broad Compatible: The usb c to usb adapter widely supports all USB C/ USB A devices like laptops, tablets, cellphones, car chargers, and phone chargers. Such as compatible with MacBook Pro/Air 2023/2022, Thunderbolt 4/3 Devices,Apple MagSafe Watch 9/8/7/SE/Ultra, iPad Pro 2022/2021, Samsung Galaxy S23/S20/S10, and iPhone 17/16/15 Pro. Plug and play
- Please Note: To reach 10Gbps speed, keep the cable under 3.3 ft. For USB A Male to USB C adapters, try flipping the USB C connector. USB C Male to USB A adapters support bidirectional 10Gbps transfer within 3.3 ft
For site owners who need managed crawling, Cloudflare announced Browser Rendering’s /crawl endpoint on March 10, 2026. The announcement describes sitemap and link discovery, HTML, Markdown, and structured JSON output, crawl depth and page-limit controls, path scoping, incremental crawling, and support for robots.txt directives including crawl delay. Cloudflare explicitly says the endpoint cannot bypass Cloudflare bot detection or captchas and identifies itself as a bot. It is an option for authorized, policy-compliant crawling—not a workaround for denial. Cloudflare Browser Rendering /crawl announcement.
Choose an execution environment that fits the page
- Server-side HTTP client: suitable when the authorized content is available in the HTTP response and you need a controlled request/session flow.
- Browser JavaScript: subject to browser-managed security rules, including the restriction on setting
Cookiedirectly. - Cloudflare Worker: follows Fetch API request semantics but has different cookie handling from a browser; carefully configure redirect behavior and cache variation.
- Managed browser rendering: consider it when permitted content depends on JavaScript execution or when you need crawl controls. It does not make blocked access permissible.
Cloudflare Workers also documents cache variation behavior: Accept and Accept-Language can be normalized in cf.vary configuration, while other headers named by an origin’s Vary response can be handled through configured actions. This is about serving the correct cached representation, not bypassing a site’s defenses. Cloudflare Workers Request documentation.
Common failures and practical fixes
| Symptom | What to check | Safer next step |
|---|---|---|
| 403 or access denied | Published policy, supported API, authentication requirements, response body, and redirects. | Use the documented route or request permission. Do not assume another browser-looking header will help. |
| Unexpected language or format | Accept, Accept-Language, and any documented content-negotiation rules. |
Request a representation your client supports and verify the returned Content-Type. |
| Cookie appears missing | Whether the client uses a cookie jar, whether a login flow is required, and whether the runtime allows direct header assignment. | Use the authorized session mechanism; in browser code, let the browser manage cookies. |
| Credentials sent to an unexpected host | Automatic redirect following and the redirect destination’s hostname. | Set an explicit redirect policy and avoid forwarding secrets across an untrusted redirect. |
| Compressed response is unreadable | Manual Accept-Encoding overrides or duplicate decompression. |
Let the HTTP library negotiate and decode compression consistently. |
| Different content from a Cloudflare-fronted site | Provider transformations between edge and origin, including the documented Accept-Encoding behavior. |
Diagnose the actual request path; do not assume the origin receives every client header unchanged. |
Or skip the browser setup
If your task is to capture a page image or PDF rather than build and maintain a browser-rendering pipeline, ScreenshotNeo is a website screenshot API and MCP server for developers. One GET request can return a PNG, JPEG, WebP, or PDF. Example using cURL (see the ScreenshotNeo API documentation):
curl -G "https://api.screenshotneo.com/v1/shot" -d access_key=YOUR_API_KEY --data-urlencode url=https://stripe.com -o shot.webpFor Python:
import requests
r = requests.get("https://api.screenshotneo.com/v1/shot", params={"access_key": "YOUR_API_KEY", "url": "https://stripe.com"}, timeout=90)
open("shot.webp", "wb").write(r.content)For Node.js:
const q = new URLSearchParams({ access_key: 'YOUR_API_KEY', url: 'https://stripe.com' });
const res = await fetch(`https://api.screenshotneo.com/v1/shot?${q}`);ScreenshotNeo accepts cookie and consent banners before capture and removes more than 60 known consent platforms, newsletter popups, and chat widgets; each of those steps can be turned off. Bot checks and CAPTCHAs, blank pages, timeouts, failed loads, and cache hits are not billed, and responses indicate the page verdict and billing status in headers. Its MCP server offers take_screenshot, get_page_info, and capture_pdf for AI agents and MCP clients. The free plan includes 1,000 screenshots per month with no card; paid plans start at $5 for 3,000 shots. This is a capture service, not authorization to access a page that denies you. Sign up for 1,000 free screenshots a month, with no card required.
Quick wins for a faster PC:
Scan for outdated or missing drivers - takes under a minuteDriver Scan →Repair Windows errors before they cause bigger problemsFix Now →Fix the driver behind crashes, sound loss and screen glitchesFind Drivers →Frequently Asked Questions
Does changing the User-Agent usually fix scraper blocks?
No universal outcome is established. A User-Agent is a declared string, not proof of identity, and you should not spoof one to evade a site’s controls.
Best Value
- 5-in-1 Connectivity: Equipped with a 4K HDMI port, a 5 Gbps USB-C data port, two 5 Gbps USB-A ports, and a USB C 100W PD-IN port. Note: The USB C 100W PD-IN port supports only charging and does not support data transfer devices such as headphones or speakers.
- Powerful Pass-Through Charging: Supports up to 85W pass-through charging so you can power up your laptop while you use the hub. Note: Pass-through charging requires a charger (not included). Note: To achieve full power for iPad, we recommend using a 45W wall charger.
- Transfer Files in Seconds: Move files to and from your laptop at speeds of up to 5 Gbps via the USB-C and USB-A data ports. Note: The USB C 5Gbps Data port does not support video output.
- HD Display: Connect to the HDMI port to stream or mirror content to an external monitor in resolutions of up to 4K@30Hz. Note: The USB-C ports do not support video output.
- What You Get: Anker 332 USB-C Hub (5-in-1), welcome guide, our worry-free 18-month warranty, and friendly customer service.
Does robots.txt give permission to scrape a site?
No. It communicates crawler preferences; it is not an access-control mechanism or a substitute for permission.
Can browser JavaScript set the Cookie header?
No. The browser manages cookies, while server runtimes may expose different behavior.
Can I use ScreenshotNeo to get around a CAPTCHA or access denial?
No. ScreenshotNeo does not make access authorized; its billing treatment for bot checks and CAPTCHAs does not mean those controls are bypassed.
