A web scraping API webhook is an HTTP callback that the provider sends to an endpoint you control when a configured job event occurs. Start the scrape, acknowledge the callback quickly, and let a queue or worker fetch and process the result. The pattern handles long-running jobs without constant polling, but event names, payloads, authentication, retry limits, and result retrieval differ by provider.
Webhook workflow in one view
- Start a job. Your application submits a URL, actor, task, or dataset request.
- Configure an event. You select a success, failure, completion, or other lifecycle event and provide an HTTPS request URL.
- Receive the callback. The scraping service sends an HTTP request, normally POST with JSON.
- Acknowledge promptly. Return a 2xx response after validating and durably recording the notification.
- Process asynchronously. A worker obtains the result from the provider’s documented endpoint or storage and updates your application.
The callback is a notification, not necessarily the scraped data itself. Bright Data’s documented asynchronous flow, for example, returns a snapshot identifier when a job is triggered. You monitor that snapshot, wait for a ready state, and then download the result; a notify URL can announce completion. Do not assume that a webhook body contains the full dataset.
Choose the event and scope it to the right job
Apify event configuration
Apify’s webhook creation model requires a request URL, one or more event types, and a condition. Events can be associated with an Actor run or build, so scope the condition to the Actor, task, or resource that matters to your workflow. A payload template can include the event type, event data, and the triggering resource. Templates must resolve to valid JSON.
Use separate webhooks when success and failure require different handling, or include an explicit event field and route both to one endpoint. Keep the payload minimal: include a stable job or run identifier, event type, and any provider reference your worker needs to retrieve results.
#1 Best Overall
- Read Before You Buy — No Video Output: These adapters support charging and USB 2.0 data transfer, but cannot transmit video signals. Except for standard USB webcams (which use USB data only), they are not compatible with HDMI/DisplayPort cables, video-capable USB-C hubs, or docking stations with video output.
- Convert USB-A Ports to USB-C: Designed to connect USB-C earphones, cables, flash drives, card readers, and other USB-C accessories to standard USB-A ports. Plug-and-play with no drivers or software required.
- Aluminum Alloy Housing: Built with a sturdy aluminum alloy shell that aids in heat dissipation and protects against daily wear and scratches. Designed to maintain a stable and secure connection.
- Compact & Travel-Friendly: The ultra-compact design allows the adapter to stay plugged into your device without blocking adjacent ports or adding bulk, reducing wear and tear on your original USB ports.
- 12-Month Warranty: Backed by a 12-month manufacturer warranty for peace of mind. Designed to meet strict quality control standards for reliable everyday performance.
Provider differences
| Concern | Apify (documented behavior) | Bright Data (documented asynchronous flow) |
|---|---|---|
| Starting work | Create a webhook with request URL, event types, and condition; associate it with the relevant Actor resource. | Trigger an asynchronous job and receive a snapshot ID. |
| Completion information | POST JSON can use a custom payload template and resource variables. | Check snapshot progress; states include starting, running, ready, and failed. A notify URL can announce completion. |
| Getting results | Use the result endpoint or storage mechanism documented for the triggering resource. | Download results after the snapshot reaches ready. |
| Delivery failures | Non-2xx responses are errors; exponential retries are documented, up to eleven retries. | Consult the current endpoint documentation for notify delivery and retry semantics. |
| Receiver timeout | Two-minute webhook request timeout is documented. | Provider-specific; verify current documentation. |
These behaviors are provider-specific. Before implementing another service, verify its event list, payload fields, result lookup method, acknowledgment requirement, timeout, retry schedule, duplicate-delivery policy, and signature or authentication options.
Build a receiver that is safe to retry
Request validation and fast acknowledgment
Expose an HTTPS endpoint and protect it with a secret that is not committed to source control. Apify recommends placing a secret token in the webhook URL and supports header templates; some headers are provider-controlled and may be overwritten. In addition to the provider’s mechanism, validate the HTTP method, content type, expected event type, and a known resource or job identifier.
Do not run a large scrape-result download, database migration, or email send inside the request handler. Parse and validate the body, write a durable queue record, and return a 2xx response. Apify documents a two-minute timeout; a slow handler can cause a delivery to be treated as failed even if the eventual work succeeds.
Node.js receiver example
import express from "express";
import crypto from "node:crypto";
const app = express();
app.use(express.json({ limit: "256kb" }));
const WEBHOOK_TOKEN = process.env.WEBHOOK_TOKEN;
app.post("/webhooks/scraping", async (req, res) => {
if (req.get("authorization") !== `Bearer ${WEBHOOK_TOKEN}`) {
return res.sendStatus(401);
}
const { eventType, jobId, resource } = req.body;
if (!eventType || !jobId || !resource) return res.sendStatus(400);
const deliveryKey = req.get("x-delivery-id") ||
crypto.createHash("sha256").update(JSON.stringify(req.body)).digest("hex");
// Insert deliveryKey with a unique constraint, then enqueue if new.
await enqueueOnce({ deliveryKey, eventType, jobId, resource });
return res.sendStatus(202);
});
app.listen(process.env.PORT || 3000);enqueueOnce must be durable and atomic. A database row with a unique deliveryKey, or a queue with a deduplication key, prevents two workers from starting the same downstream action.
What’s actually slowing this PC down?
Pick the symptom - the matching free tool is one click away.
Rank #2
- 5-in-1 USB-C Hub: Experience comprehensive connectivity featuring a Power Delivery input, two USB-A 2.0 ports, a USB-A 3.0 port, and an HDMI port. (Note: The USB-C power delivery input port is only for connecting an external wall charger to power your laptop and cannot power peripheral devices.)
- 90W Pass-Through Charging: Achieve optimal charging with 90W pass-through power to your laptop, supported by a total input of 100W, with the hub reserving 10W for operational efficiency. (Note: Wall charger not included.)
- Quick Data Transfers: Accelerate your productivity with rapid data transfers using a high-speed 5Gbps USB 3.0 port and two 480Mbps USB 2.0 ports.
- 4K HDMI Display: Enhance your visual experience with a hub capable of delivering 4K resolution at 30Hz in both mirror and extend modes. Please note that this hub is compatible with MacBook (macOS 12 and newer), Windows 10 and 11, ChromeOS, and laptops equipped with DP Alt Mode and Power Delivery. Note: This device is not compatible with Linux.
- What You Get: Anker USB-C Hub (5-in-1, 4K HDMI), welcome guide, 18-month warranty, and our friendly customer service.
Python receiver example
import hashlib
import json
import os
from flask import Flask, request, abort
app = Flask(__name__)
TOKEN = os.environ["WEBHOOK_TOKEN"]
@app.post("/webhooks/scraping")
def scraping_webhook():
if request.headers.get("Authorization") != f"Bearer {TOKEN}":
abort(401)
data = request.get_json(silent=True) or {}
for field in ("eventType", "jobId", "resource"):
if not data.get(field):
abort(400)
delivery_id = request.headers.get("X-Delivery-Id") or hashlib.sha256(
json.dumps(data, sort_keys=True).encode()
).hexdigest()
enqueue_once(delivery_id, data) # durable insert with a unique key
return ("", 202)Return a success code only after the notification is safely recorded. If your queue is unavailable, return a non-2xx response so a provider that supports retries can try again.
Handle retries and duplicate notifications
Apify says non-2xx responses trigger exponential backoff and documents up to eleven retries; the eleventh retry occurs approximately 32 hours after the initial attempt. It also warns: “In rare cases, the webhook might be invoked more than once. Design your code to be idempotent to handle duplicate calls.” These numbers describe Apify, not a universal scraping-API guarantee.
Use two different idempotency controls
- Webhook creation idempotency: Apify accepts an idempotency key when creating a webhook, preventing repeated create requests from producing duplicate webhook records.
- Delivery processing idempotency: Your receiver still needs a unique delivery, event, or job key. The creation key does not deduplicate incoming callbacks.
For a success event, use an upsert such as “set job status to complete” rather than “insert a completion row.” For a failure event, store the provider’s error details and allow a later success event to advance the state if the provider documents that sequence. If no stable delivery ID is supplied, derive a key from immutable fields such as provider resource ID, event type, and event timestamp; avoid hashing mutable fields or the entire body when it can change between retries.
Fetch the result after notification
- Read the job or snapshot identifier from the validated payload.
- Ask the provider’s status endpoint for the current state when the callback means “ready to check,” rather than assuming readiness.
- On a ready state, download the result using the provider’s authenticated endpoint or storage reference.
- Write the result to durable storage with a unique job key.
- Record processing time, provider response code, result location, and any error for replay and support.
For Bright Data’s flow, the snapshot progresses through starting, running, ready, or failed; use the snapshot ID to monitor and retrieve data. For Apify, follow the result endpoint or storage reference associated with the triggering resource. Keep credentials in a secret manager and send authorization only to the provider host.
Rank #3
- Sleek 7-in-1 USB-C Hub: Features an HDMI port, two USB-A 3.0 ports, and a USB-C data port, each providing 5Gbps transfer speeds. It also includes a USB-C PD input port for charging up to 100W and dual SD and TF card slots, all in a compact design.
- Flawless 4K@60Hz Video with HDMI: Delivers exceptional clarity and smoothness with its 4K@60Hz HDMI port, making it ideal for high-definition presentations and entertainment. (Note: Only the HDMI port supports video projection; the USB-C port is for data transfer only.)
- Double Up on Efficiency: The two USB-A 3.0 ports and a USB-C port support a fast 5Gbps data rate, significantly boosting your transfer speeds and improving productivity.
- Fast and Reliable 85W Charging: Offers high-capacity, speedy charging for laptops up to 85W, so you spend less time tethered to an outlet and more time being productive.
- What You Get: Anker USB-C Hub (7-in-1), welcome guide, 18-month warranty, and our friendly customer service.
Security checklist for callback endpoints
- Require HTTPS and reject unexpected methods and content types.
- Use a high-entropy token or provider-supported signature; rotate it without exposing it in logs.
- Allow-list provider IP ranges only if the provider publishes and maintains them; do not treat an IP list as a replacement for authentication.
- Limit body size and parsing depth to reduce denial-of-service risk.
- Validate that the event belongs to a job your system created.
- Store only the fields your worker needs; avoid putting scraped personal data in a URL or log line.
- Redact authorization headers and tokens from request logs.
- Use a dead-letter queue and an operator replay procedure for malformed or permanently failed deliveries.
Testing, observability, and operations
Test the important paths
- Valid success event returns 202 and creates one queue item.
- Invalid token returns 401 and creates no work.
- Malformed JSON or unknown job returns 400.
- Two identical deliveries create one downstream action.
- A simulated queue outage returns non-2xx and is retried.
- A provider failure event records the failure without attempting a result download.
- A result endpoint that temporarily returns an error is retried by your worker with a bounded backoff.
Measure delivery and processing separately
Track callback count, acknowledgment latency, non-2xx responses, deduplication count, queue age, result-download latency, and final job state. Alert on a growing dead-letter queue or jobs that remain running beyond the provider’s normal window. Keep a correlation ID from the initial scrape request through the webhook, worker, and result download.
Common failures and fixes
The provider reports a timeout
Your handler is doing slow work before responding. Validate, enqueue, and acknowledge immediately; move downloads and transformations to a worker.
The same job runs twice
Retries or duplicate dispatches are reaching a non-idempotent handler. Add a database uniqueness constraint or queue deduplication key and make state updates repeat-safe.
Every callback is unauthorized
Check whether the provider sends a URL token, a header, or a signature. Confirm proxy configuration preserves the expected header and compare the secret from your secret manager, not a copied log value.
Quick wins for a faster PC:
Clear out junk files and repair common Windows errorsFree Scan →Fix the driver behind crashes, sound loss and screen glitchesFind Drivers →Repair Windows errors before they cause bigger problemsFix Now →Rank #4
- Dual Converters, Infinite Potential：Includes 2× USB C male to USB A female adapters and 2× USB A male to USB C female adapters. Perfect for a wide range of uses—tablets with Bluetooth keyboards, expand USB ports on macbook, and more. Two different converters for all your daily needs
- Next-Level 10Gbps & 3A Charging: No more slow 480Mbps, this usb to usb c adapter has a transfer speed of up to 10Gbps, allowing you to do more transferring in less time. This usb adapter fits both USB A and USB C charger, supporting up to 3A fast charging
- Upgraded Exquisite Craftsmanship: With an aluminum alloy housing and metal connector, the usbc to usb adapter is extremely durable and sturdy. Rigorously tested to withstand more than 10,000 times of plugging and unplugging, ensuring long-lasting performance
- Broad Compatible: The usb c to usb adapter widely supports all USB C/ USB A devices like laptops, tablets, cellphones, car chargers, and phone chargers. Such as compatible with MacBook Pro/Air 2023/2022, Thunderbolt 4/3 Devices,Apple MagSafe Watch 9/8/7/SE/Ultra, iPad Pro 2022/2021, Samsung Galaxy S23/S20/S10, and iPhone 17/16/15 Pro. Plug and play
- Please Note: To reach 10Gbps speed, keep the cable under 3.3 ft. For USB A Male to USB C adapters, try flipping the USB C connector. USB C Male to USB A adapters support bidirectional 10Gbps transfer within 3.3 ft
You receive completion but no data
A notification can be separate from the result. Use the supplied job or snapshot ID, check status, and call the documented result endpoint after readiness.
Creating the webhook creates several records
Retries in your provisioning code may have repeated the create request. Use the provider’s webhook-creation idempotency key where available, then remove accidental duplicates carefully.
Independent reader supportYour contribution helps us test, update, and keep practical guides available for everyone.Or skip the browser setup
If your actual task is taking clean screenshots rather than scraping structured records, ScreenshotNeo provides a one-call website screenshot API and MCP server. It removes cookie and consent banners, newsletter popups, and chat widgets before capture; bot checks, blank pages, failed loads, timeouts, and cache hits are not billed, and response headers identify the page verdict and billing result.
Use the API directly (see the ScreenshotNeo documentation):
Do these 3 things before closing this tab:
1Clear out junk files and repair common Windows errors2Scan for outdated or missing drivers - takes under a minute3Repair Windows errors before they cause bigger problemscurl -G "https://api.screenshotneo.com/v1/shot" -d access_key=YOUR_API_KEY --data-urlencode url=https://stripe.com -o shot.webpPython:
import requests
r = requests.get("https://api.screenshotneo.com/v1/shot", params={"access_key": "YOUR_API_KEY", "url": "https://stripe.com"}, timeout=90)
open("shot.webp", "wb").write(r.content)Node.js:
const q = new URLSearchParams({ access_key: 'YOUR_API_KEY', url: 'https://stripe.com' });
const res = await fetch(`https://api.screenshotneo.com/v1/shot?${q}`);ScreenshotNeo also has an MCP server with take_screenshot, get_page_info, and capture_pdf for AI agents. The Free plan includes 1,000 screenshots each month without a card; paid plans start at $5 for 3,000. Create a free ScreenshotNeo account.
Best Value
- 5-in-1 Connectivity: Equipped with a 4K HDMI port, a 5 Gbps USB-C data port, two 5 Gbps USB-A ports, and a USB C 100W PD-IN port. Note: The USB C 100W PD-IN port supports only charging and does not support data transfer devices such as headphones or speakers.
- Powerful Pass-Through Charging: Supports up to 85W pass-through charging so you can power up your laptop while you use the hub. Note: Pass-through charging requires a charger (not included). Note: To achieve full power for iPad, we recommend using a 45W wall charger.
- Transfer Files in Seconds: Move files to and from your laptop at speeds of up to 5 Gbps via the USB-C and USB-A data ports. Note: The USB C 5Gbps Data port does not support video output.
- HD Display: Connect to the HDMI port to stream or mirror content to an external monitor in resolutions of up to 4K@30Hz. Note: The USB-C ports do not support video output.
- What You Get: Anker 332 USB-C Hub (5-in-1), welcome guide, our worry-free 18-month warranty, and friendly customer service.
FAQ
How do I get notified when a web scraping API job is finished?
Configure the provider’s completion event with an HTTPS request URL, then have your endpoint enqueue work and acknowledge the callback. The notification may contain an identifier rather than the scraped result.
Should a webhook endpoint return 200 or 202?
Either is commonly appropriate when the notification has been accepted. Use the status code your provider documents; the important rule is to return a 2xx only after durable recording, not after all slow processing has completed.
Can I replace webhooks with polling?
Polling is a fallback when callbacks are unavailable or unreliable, but it adds repeated requests and delay. A resilient design can retain a low-frequency reconciliation poll for jobs whose callback was missed.
The Tool Desk
Outbyte Driver Updater FREEScan for outdated or missing drivers - takes under a minuteDriver Scan →Outbyte PC Repair FREEClear out junk files and repair common Windows errorsFree Scan →Frequently Asked Questions
How do I get notified when a web scraping API job is finished?
Configure a completion event and HTTPS callback URL, acknowledge it quickly, and use the job identifier to retrieve results.
How do I handle webhook retries from a scraping API?
Return non-2xx only when the notification was not durably accepted, and process accepted events idempotently with a unique delivery or job key.
Is a webhook the scraped data itself?
Not necessarily. Many asynchronous APIs send a job or snapshot identifier; retrieve the data separately after the provider reports readiness.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
