Recommended Free Tools
In a web scraping API, an HTTP status code tells you what happened at one responding layer—not necessarily whether the target page was retrieved correctly. The API may expose its own validation result, a proxy response, or the target site’s response. Treat every code together with its response headers, body, provider documentation, and billing rules. A 200 OK can still contain a CAPTCHA, login page, or empty result, while a 429 may indicate either the target site’s rate limit or your scraping plan’s concurrency limit.
This guide explains the standard meanings, shows how to identify the failing layer, and provides a practical recovery sequence for production scrapers.
Why one status code can describe three different failures
RFC 9110 defines HTTP semantics, but a scraping API adds an implementation layer. Your request first reaches the scraping service. That service may call a proxy, which calls the target website, then return a transformed response to you. The code you see can therefore belong to:
- The scraping API: for example, an invalid API key, malformed parameter, plan limit, or internal failure.
- The proxy: such as proxy authentication failure (407) or a proxy connection error.
- The target website: such as a target 403, 404, or 429.
Providers differ in whether they pass through the target’s code, wrap it in an API error, retry first, or return a provider-specific code. Inspect the provider’s error schema and headers instead of assuming that a familiar number has a universal operational meaning. The standards reference is RFC 9110; a readable index is MDN’s HTTP status reference.
#1 Best Overall
- Read Before You Buy — No Video Output: These adapters support charging and USB 2.0 data transfer, but cannot transmit video signals. Except for standard USB webcams (which use USB data only), they are not compatible with HDMI/DisplayPort cables, video-capable USB-C hubs, or docking stations with video output.
- Convert USB-A Ports to USB-C: Designed to connect USB-C earphones, cables, flash drives, card readers, and other USB-C accessories to standard USB-A ports. Plug-and-play with no drivers or software required.
- Aluminum Alloy Housing: Built with a sturdy aluminum alloy shell that aids in heat dissipation and protects against daily wear and scratches. Designed to maintain a stable and secure connection.
- Compact & Travel-Friendly: The ultra-compact design allows the adapter to stay plugged into your device without blocking adjacent ports or adding bulk, reducing wear and tear on your original USB ports.
- 12-Month Warranty: Backed by a 12-month manufacturer warranty for peace of mind. Designed to meet strict quality control standards for reliable everyday performance.
HTTP status classes at a glance
| Class | Range | General meaning | Scraping interpretation |
|---|---|---|---|
| 1xx | 100–199 | Informational | Intermediate protocol signals; most scraping clients do not expose them as final results. |
| 2xx | 200–299 | Successful response | Transport succeeded at the responding layer; validate the body before accepting scraped data. |
| 3xx | 300–399 | Redirection | Check whether the client or provider followed redirects and inspect the final response. |
| 4xx | 400–499 | Client or access error | Could be your request, credentials, rate, permissions, or a target resource problem. |
| 5xx | 500–599 | Server error | Identify whether the API, proxy, or target server failed before retrying. |
Common codes and the sensible next action
200 OK: successful transport, unproven content
200 means the request was successful at the layer that sent the response. It does not prove that the body contains the intended article, product record, or HTML. A target can return a CAPTCHA, bot-check page, login form, error template, or empty shell with status 200. ScraperAPI documents CAPTCHA detection and retry behavior for its own service; that behavior is not a universal rule for all APIs.
Validate at least the content type, minimum body length, page title, expected selector or JSON field, and a block-page signature. Save the response body when validation fails so you can distinguish a target redesign from an access challenge.
301, 302, and other 3xx redirects
A redirect describes the response you received, not necessarily the page you wanted. Verify the provider’s redirect-following setting and inspect the final URL and final status. A redirect to a login page, consent page, regional domain, or CAPTCHA endpoint should be treated as a content-validation failure even if the final transport response is 200. Redirect semantics and method handling are defined by RFC 9110.
400 Bad Request
Usually the API rejected a malformed or unsupported request. Check URL encoding, required parameters, mutually exclusive options, and value formats. ScraperAPI labels 400 as a malformed request and specifically advises checking the URL. Do not retry an unchanged malformed request; fix it and log the provider’s error body.
The Tool Desk
Outbyte Driver Updater FREEFix the driver behind crashes, sound loss and screen glitchesFind Drivers →Outbyte PC Repair FREEClear out junk files and repair common Windows errorsFree Scan →Rank #2
- 5-in-1 USB-C Hub: Experience comprehensive connectivity featuring a Power Delivery input, two USB-A 2.0 ports, a USB-A 3.0 port, and an HDMI port. (Note: The USB-C power delivery input port is only for connecting an external wall charger to power your laptop and cannot power peripheral devices.)
- 90W Pass-Through Charging: Achieve optimal charging with 90W pass-through power to your laptop, supported by a total input of 100W, with the hub reserving 10W for operational efficiency. (Note: Wall charger not included.)
- Quick Data Transfers: Accelerate your productivity with rapid data transfers using a high-speed 5Gbps USB 3.0 port and two 480Mbps USB 2.0 ports.
- 4K HDMI Display: Enhance your visual experience with a hub capable of delivering 4K resolution at 30Hz in both mirror and extend modes. Please note that this hub is compatible with MacBook (macOS 12 and newer), Windows 10 and 11, ChromeOS, and laptops equipped with DP Alt Mode and Power Delivery. Note: This device is not compatible with Linux.
- What You Get: Anker USB-C Hub (5-in-1, 4K HDMI), welcome guide, 18-month warranty, and our friendly customer service.
401 Unauthorized
RFC 9110 defines 401 as a request lacking valid authentication credentials for the target resource. A scraping provider may also use 401 for its own API-key failure; ScraperAPI lists an invalid API key as one cause. Determine which layer emitted the response before rotating credentials. Confirm the key is present, active, correctly scoped, and sent in the documented parameter or header.
403 Forbidden
403 means access was refused and is not interchangeable with 401. Supplying credentials may not help when the target or provider applies bot, geography, account, or policy restrictions. ScraperAPI notes that protected domains may require a premium request option, but that is provider-specific advice. Check the response body, target rules, account permissions, and the provider’s supported access options; avoid hammering the same URL.
404 Not Found
The requested resource was not found at the responding layer. Check the URL spelling, encoding, host, path, and whether the target resource was deleted or moved. ScraperAPI counts 404 among successful requests for billing purposes, illustrating why “successful request” can mean the API completed a request rather than that content was found.
407 Proxy Authentication Required
407 is authentication required by a proxy, distinct from 401 authentication for the target resource. Verify proxy credentials, proxy URL syntax, and whether your provider—not your own network proxy—generated the response. RFC 9110 and MDN describe the distinction.
Rank #3
- Sleek 7-in-1 USB-C Hub: Features an HDMI port, two USB-A 3.0 ports, and a USB-C data port, each providing 5Gbps transfer speeds. It also includes a USB-C PD input port for charging up to 100W and dual SD and TF card slots, all in a compact design.
- Flawless 4K@60Hz Video with HDMI: Delivers exceptional clarity and smoothness with its 4K@60Hz HDMI port, making it ideal for high-definition presentations and entertainment. (Note: Only the HDMI port supports video projection; the USB-C port is for data transfer only.)
- Double Up on Efficiency: The two USB-A 3.0 ports and a USB-C port support a fast 5Gbps data rate, significantly boosting your transfer speeds and improving productivity.
- Fast and Reliable 85W Charging: Offers high-capacity, speedy charging for laptops up to 85W, so you spend less time tethered to an outlet and more time being productive.
- What You Get: Anker USB-C Hub (7-in-1), welcome guide, 18-month warranty, and our friendly customer service.
429 Too Many Requests
429 signals excessive requests. It may come from the target, the provider’s account limiter, or a concurrency quota. ScraperAPI documents excessive simultaneous requests and recommends checking plan concurrency. Reduce parallelism, honor any Retry-After header, add exponential backoff with jitter, and review per-minute and concurrent-request limits. Retrying immediately at the same rate generally makes the condition worse.
5xx server errors
5xx is a server-error class, not proof that the target is at fault. Capture the provider request ID and determine whether the API, proxy, or target generated it. Retry only transient failures according to the provider’s policy, with bounded exponential backoff and an idempotent request design. ScraperAPI states that requests failing after 70 seconds of retrying are not charged; do not apply that timing or billing rule to another service.
A repeatable diagnostic workflow
- Record the event. Store timestamp, requested URL, status, final URL, response headers, response body (subject to privacy policy), provider request ID, elapsed time, and your plan/concurrency state.
- Locate the responding layer. Use provider-specific error fields, headers, nested target-status fields, and documentation. A plain pass-through 403 is operationally different from an API-generated 403.
- Validate successful bodies. For 200–299 responses, check content type, expected fields or selectors, title, canonical URL, body size, and known CAPTCHA/login/error markers.
- Apply code-specific checks. For 400 verify syntax; 401 and 407 verify the appropriate credentials; 403 verify permission and access policy; 404 verify resource existence; 429 verify rate and concurrency.
- Retry selectively. Retry transient 5xx and provider-documented temporary failures. Do not retry unchanged 400, invalid 401, permanent 403, or confirmed 404. Back off on 429.
- Track billing separately. “HTTP success,” “content success,” and “billable request” are different dimensions. Read the current provider policy for 200, 404, retries, cache hits, and failures.
Minimal client-side validation example
The following Python pattern treats HTTP success and data success as separate decisions. Adapt the field names and block-page tests to your target and provider.
import requests
url = "https://example.com/article"
r = requests.get(url, timeout=60, allow_redirects=True)
print("status:", r.status_code)
print("final_url:", r.url)
print("content_type:", r.headers.get("content-type"))
if r.status_code == 429:
raise RuntimeError("Rate limited; reduce rate/concurrency and back off")
if r.status_code in (401, 407):
raise RuntimeError("Check API, target, or proxy credentials")
if r.status_code >= 400:
raise RuntimeError(f"HTTP failure: {r.status_code}")
body = r.text.lower()
blocked = any(marker in body for marker in ("captcha", "verify you are human", "log in"))
if blocked or len(body) < 1000:
raise RuntimeError("Transport succeeded, but expected content was not returned")
When using a scraping API, preserve the provider’s raw response metadata before parsing so an apparently valid 200 can be investigated later.
Crashes, No Sound, or Screen Glitches?
Random freezes, missing sound and display glitches usually trace back to one bad driver. Find and replace yours safely.Free scan · under a minutePC Slower Than It Used to Be?
A free scan shows the junk files, broken settings and background clutter dragging Windows down - then fixes them in one click.Free scan · Windows 10 & 11Rank #4
- Dual Converters, Infinite Potential：Includes 2× USB C male to USB A female adapters and 2× USB A male to USB C female adapters. Perfect for a wide range of uses—tablets with Bluetooth keyboards, expand USB ports on macbook, and more. Two different converters for all your daily needs
- Next-Level 10Gbps & 3A Charging: No more slow 480Mbps, this usb to usb c adapter has a transfer speed of up to 10Gbps, allowing you to do more transferring in less time. This usb adapter fits both USB A and USB C charger, supporting up to 3A fast charging
- Upgraded Exquisite Craftsmanship: With an aluminum alloy housing and metal connector, the usbc to usb adapter is extremely durable and sturdy. Rigorously tested to withstand more than 10,000 times of plugging and unplugging, ensuring long-lasting performance
- Broad Compatible: The usb c to usb adapter widely supports all USB C/ USB A devices like laptops, tablets, cellphones, car chargers, and phone chargers. Such as compatible with MacBook Pro/Air 2023/2022, Thunderbolt 4/3 Devices,Apple MagSafe Watch 9/8/7/SE/Ultra, iPad Pro 2022/2021, Samsung Galaxy S23/S20/S10, and iPhone 17/16/15 Pro. Plug and play
- Please Note: To reach 10Gbps speed, keep the cable under 3.3 ft. For USB A Male to USB C adapters, try flipping the USB C connector. USB C Male to USB A adapters support bidirectional 10Gbps transfer within 3.3 ft
Retries, backoff, and reliability design
Use bounded exponential backoff
For transient 429 or 5xx responses, wait progressively longer (for example, 1, 2, 4, then 8 seconds) with random jitter, and cap both attempts and total elapsed time. Prefer a provider-supplied Retry-After value when present. Do not retry non-idempotent operations unless the API documents them as safe.
Separate queues by failure type
Keep rate-limited jobs from blocking malformed requests. A queue for retryable transport failures, a quarantine queue for content-validation failures, and a permanent-failure log make recovery observable and prevent infinite loops.
Measure content quality
Monitor field completeness, duplicate pages, unexpected redirects, body-size shifts, CAPTCHA markers, and parse-error rates—not only HTTP status distributions. A rise in 200 responses with missing fields often indicates blocking or a target redesign.
Independent reader supportYour contribution helps us test, update, and keep practical guides available for everyone.Comparing scraping APIs by status-code behavior
Before switching providers, ask these concrete questions:
Best Value
- 5-in-1 Connectivity: Equipped with a 4K HDMI port, a 5 Gbps USB-C data port, two 5 Gbps USB-A ports, and a USB C 100W PD-IN port. Note: The USB C 100W PD-IN port supports only charging and does not support data transfer devices such as headphones or speakers.
- Powerful Pass-Through Charging: Supports up to 85W pass-through charging so you can power up your laptop while you use the hub. Note: Pass-through charging requires a charger (not included). Note: To achieve full power for iPad, we recommend using a 45W wall charger.
- Transfer Files in Seconds: Move files to and from your laptop at speeds of up to 5 Gbps via the USB-C and USB-A data ports. Note: The USB C 5Gbps Data port does not support video output.
- HD Display: Connect to the HDMI port to stream or mirror content to an external monitor in resolutions of up to 4K@30Hz. Note: The USB-C ports do not support video output.
- What You Get: Anker 332 USB-C Hub (5-in-1), welcome guide, our worry-free 18-month warranty, and friendly customer service.
- Which layer’s status is surfaced, and are target and provider errors distinguishable?
- Does the service inspect bodies for CAPTCHAs, login pages, or other blocked content?
- What are the documented retry, timeout, rate, and concurrency rules?
- Are 200, 404, cached responses, and failed retries billed differently?
- Does every error include a stable code, request ID, and actionable message?
Standards explain semantics, but they do not establish a universal success rate, billing model, or retry guarantee. Compare current provider documentation on those dimensions rather than inferring behavior from the numeric code alone.
Or skip the browser setup
If your task is to obtain clean visual captures rather than parse records, ScreenshotNeo provides a website screenshot API and MCP server. A single request returns PNG, JPEG, WebP, or PDF, and the service accepts cookie/consent banners before removing more than 60 known consent platforms, newsletter popups, and chat widgets. Bot checks, blank pages, timeouts, failed loads, and cache hits are not billed; response headers identify the page verdict and billing result.
cURL:
curl -G "https://api.screenshotneo.com/v1/shot" -d access_key=YOUR_API_KEY --data-urlencode url=https://stripe.com -o shot.webpPython:
import requests
r = requests.get("https://api.screenshotneo.com/v1/shot", params={"access_key": "YOUR_API_KEY", "url": "https://stripe.com"}, timeout=90)
open("shot.webp", "wb").write(r.content)Node.js:
const q = new URLSearchParams({ access_key: 'YOUR_API_KEY', url: 'https://stripe.com' });
const res = await fetch(`https://api.screenshotneo.com/v1/shot?${q}`);See the complete options and response details in the ScreenshotNeo documentation. Its MCP server includes take_screenshot, get_page_info, and capture_pdf tools for Claude, Cursor, and other MCP clients. Plans include 1,000 screenshots per month free with no card; paid plans start at $5 for 3,000 shots. Create a free ScreenshotNeo account.
Google crawler codes are a different context
Google documents that 429 and 5xx responses cause its crawlers to temporarily slow crawling, and that a 2xx response does not guarantee indexing. Those statements apply to Google’s crawling and indexing systems, not to scraping APIs or every HTTP client. Do not use them as a general retry or success rule for your scraper. See Google’s HTTP status troubleshooting documentation.
Do these 3 things before closing this tab:
1Repair Windows errors before they cause bigger problems2Fix the driver behind crashes, sound loss and screen glitches3Clear out junk files and repair common Windows errorsFrequently Asked Questions
Should I retry every non-200 response from a scraping API?
No. Retry only documented transient failures, typically selected 429 and 5xx cases, using bounded backoff. Correct malformed 400 requests and credentials or permissions for 401, 403, and 407 instead of repeating them.
How can I tell whether a 403 came from the target or the scraping provider?
Inspect provider-specific error fields, headers, request IDs, and documentation. A nested target-status field or pass-through body suggests the target; an API error schema usually indicates provider handling.
Can a 404 be billable even though no page was found?
Yes. Billing is provider-specific. ScraperAPI documents 404 as a successful request for billing, so content existence and request billing must be tracked separately.
What should I store for debugging status-code failures?
Store the timestamp, requested and final URLs, status, headers, provider request ID, response body where permitted, elapsed time, retry count, and rate/concurrency state.
What’s actually slowing this PC down?
Pick the symptom - the matching free tool is one click away.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
