If Apify requests are being blocked, first determine whether the proxy connection is failing or the target site is rejecting the request. Test Apify Proxy independently, inspect the target’s response, then choose a proxy group and session behavior that fit the job. A proxy can change the route and apparent IP; it cannot guarantee access through every anti-scraping system.
First identify what is failing
A proxy error and a target-site block are different failures. A proxy connection can fail before a request reaches the site; a site block means the request got far enough for the target to return a denial, challenge, or incomplete page.
-
Test the proxy connection with Apify’s documented proxy status endpoint: https://proxy.apify.com/?format=json. An error here points to proxy configuration, credentials, connectivity, or account access rather than the target site.
-
Check what IP the request appears to use by sending a request through the proxy to Apify’s browser-info endpoint: https://api.apify.com/v2/browser-info. Compare the reported result with the proxy group and country you configured.
Do these 3 things before closing this tab:
1Clear out junk files and repair common Windows errors2Fix the driver behind crashes, sound loss and screen glitches3Repair Windows errors before they cause bigger problemsSpecial offer. See more information about Outbyte and uninstall instructions. Please review EULA and Privacy policy. -
Inspect the failed target response or rendered page. Look for a status code, block message, CAPTCHA, consent screen, blank content, or content missing only after JavaScript execution. Preserve the response body and relevant logs; the visible symptom helps separate a target challenge from a network or crawler problem.
Apify Documentation describes its proxy service as letting users rotate IP addresses “when scraping to avoid geographic blocking.” That describes a capability, not a guarantee that a particular site will allow a request.
Configure Apify Proxy for an Actor
For an Actor running on Apify, use the Apify SDK proxy configuration rather than manually building a proxy URL. The SDK configuration can be passed to the crawler, where the SDK handles proxy assignment according to the selected group and session settings. Exact crawler constructor options can differ by crawler type and SDK version, so use the matching current SDK reference.
JavaScript
import { Actor } from 'apify';
import { CheerioCrawler } from 'crawlee';
await Actor.init();
const proxyConfiguration = await Actor.createProxyConfiguration({
groups: ['RESIDENTIAL'],
countryCode: 'US',
});
const crawler = new CheerioCrawler({
proxyConfiguration,
async requestHandler({ request, $, log }) {
log.info(`Fetched ${request.url}: ${$('title').text()}`);
},
});
await crawler.run(['https://example.com']);
await Actor.exit();This example selects residential proxies and the US. Replace the group and country with options available to your account and suitable for the site and task. For a default datacenter group, use the documented auto group value where supported by your SDK configuration.
Python
from apify import Actor
from crawlee.crawlers import BeautifulSoupCrawler
async def main():
async with Actor:
proxy_configuration = await Actor.create_proxy_configuration(
groups=['RESIDENTIAL'],
country_code='US',
)
crawler = BeautifulSoupCrawler(
proxy_configuration=proxy_configuration,
)
@crawler.router.default_handler
async def handle(context):
title = context.soup.title.string if context.soup.title else ''
context.log.info(f"Fetched {context.request.url}: {title}")
await crawler.run(['https://example.com'])Check the SDK reference for the version installed in your Actor if a method name, argument, or crawler constructor differs. Apify’s SDK proxy references are available for JavaScript and Python.
Choose a proxy group for the observed failure
Apify documents datacenter, residential, and Unblocker proxy options. Group availability and account access depend on the Apify account. Pick based on what the target returns, whether the workflow needs a stable session, geography, latency and billing model—not on an assumption that one group will defeat all protections.
| Option | When it may fit | Trade-offs to check |
|---|---|---|
| Datacenter | Ordinary proxy use where speed and relatively low cost are priorities. | Apify describes health checks, rotation, country selection, persistent sessions, and shared or dedicated groups. Shared IP activity by other users can contribute to blocking. |
| Residential | A task for which residential IP characteristics and the account terms are appropriate. | Pricing is traffic-based. Speed can vary with host devices, and a session can end if a device disconnects. |
Unblocker (groups-UNBLOCKER) |
A managed option when Apify’s documented handling of common protections and CAPTCHA challenges matches the task. | Uses unit-based billing and does not provide a session parameter. Availability and current pricing can change; check Apify’s current documentation and account interface before use. |
| Your own proxy URLs | A configuration alternative when you have proxy endpoints you are authorized to use and need to supply them to an Actor or Console task. | Configuration support does not establish that the provider will work against a particular site. Follow Apify’s current custom-proxy format and your provider’s credential and usage requirements. |
Apify’s official references include the proxy overview, datacenter proxy details, and residential proxy details. Check the current account UI for group eligibility and billing; the available evidence does not establish one universal price for these options.
Set up a proxy in an external client
If the crawler does not run as an Apify Actor, use Apify’s documented proxy host, port, username parameters, and proxy password in the client’s proxy settings. The documented hostname and port are proxy.apify.com:8000. Username parameters select applicable options such as a group, country, or session; use the exact syntax and option names in Apify’s current proxy documentation.
Do not paste a real proxy password into source code, logs, screenshots, or a public repository. Store credentials in environment variables or a secrets manager. Apify warns that proxy usage is charged to the account and that the HTTP proxy password is sent unencrypted by the HTTP protocol; protect the connection and credentials accordingly.
Rank #2
- # Unlimited Bandwidth to use
- # Endless list of countries to connect to worldwide!
- # Simple one click to connect
- # Super fast speed proxy
- # Proxy any apps and sites in any country
Apify also documents using customer-supplied proxy URLs in Actors and Console tasks. This is a setup option, not proof that a different provider will avoid a specific target’s controls. See Apify’s instructions for using your own proxy.
Choose whether to keep or rotate the IP
Use a session when continuity matters
A session keeps requests associated with the same IP for continuity. This can suit a workflow that needs to preserve a login or other state across requests. Where the crawler also manages cookies or browser fingerprints, Apify’s SessionPool can coordinate session and cookie/fingerprint management. A stable session is not a promise that the site will accept the traffic.
Rotate when a different IP is appropriate
Rotation changes the IP used for requests. It may be useful when a task calls for changing IPs, but rapid rotation is not a substitute for reasonable request rates or authorization. Avoid tight retry loops: they can amplify load and make diagnosis harder.
Recommended Free Tools
Account for browser and crawler lifecycle
A browser-based crawler can retain an assigned IP across multiple requests depending on how its proxy configuration, sessions, and browser lifecycle are set up. Do not assume that each page request automatically gets a new IP. Inspect the SDK crawler behavior and session configuration for the version and crawler type you run.
Unblocker does not provide a session parameter, so do not select it for a workflow whose design depends on that parameter. Consult the JavaScript proxy configuration reference or Python proxy management reference for current session options.
Interpret proxy errors before retrying
Apify documents proxy response codes in the 590–599 range. The status identifies a proxy-side issue; it is distinct from an HTTP response generated by the target. The examples below are useful diagnostics, not a complete list of every failure.
| Code | Documented meaning | What to check |
|---|---|---|
| 593 | DNS or configuration error | Check the proxy hostname, URL syntax, selected options, and whether the target hostname resolves as expected. |
| 594–596 | Connection, refusal, or reset problems | Check connectivity, the configured endpoint and port, and whether the failure is transient. Do not repeatedly retry at high frequency. |
| 597 | Upstream authentication failed | Check the proxy password, username parameter syntax, and account access to the selected group. |
| 599 | Generic upstream error | Review the full error and logs, then test the proxy endpoint separately to determine whether the issue is proxy-side or target-side. |
For the current list and descriptions, see Apify’s proxy error codes. A target’s CAPTCHA, access-denied page, or challenge is not itself evidence that proxy authentication failed.
The Tool Desk
Outbyte Driver Updater FREEFix the driver behind crashes, sound loss and screen glitchesFind Drivers →Outbyte PC Repair FREERepair Windows errors before they cause bigger problemsFix Now →Common problems and fixes
The status endpoint fails
Confirm that the Actor or external client can reach the documented proxy host and port, that credentials are valid, and that the chosen group is available to the account. If the status endpoint reports a proxy error, resolve that before changing crawler selectors or target-site handling.
The proxy works, but the target returns a block page
Verify that the request is actually routed through the intended group and country, then review whether the task needs a stable session or a different group. A proxy only changes connection routing and IP characteristics; anti-scraping systems can use other signals. Do not treat repeated retries or changing IPs as a guaranteed solution.
Some content is missing or the page is blank
Determine whether the target returned an error or whether the crawler failed to wait for or execute the page’s required scripts. For browser crawlers, inspect the relevant navigation, wait, and request logs. Apify provides a guide to blocked requests in PuppeteerCrawler; browser-specific troubleshooting may not apply to HTTP-only crawlers.
Authentication fails with 597
Check the proxy username parameter spelling and group selection against the documentation, confirm the password belongs to the expected Apify account, and verify account access. Keep the actual password out of logs and shared code.
Free tools Windows power users keep installed
One-click scans. No signup required.
Failures appear only after several pages
Check whether the workflow needs session continuity, whether the crawler is reusing or replacing sessions as intended, and whether its request pace is appropriate. If requests should share state, use session management rather than assuming each request has an independent identity.
Use proxies responsibly and account for their limits
Apify Academy’s proxy guidance discusses IP quality and rate limits while noting that anti-scraping techniques extend beyond IP blocking. No proxy group guarantees that every site will return the desired content. A proxy configuration also does not establish permission to access or collect a site’s data. Follow the site’s terms, applicable law, and the authorization that applies to your task; the behavior or policy of any particular target site is not established here.
Or skip the browser setup
If the goal is a website screenshot rather than a crawler workflow, ScreenshotNeo is a screenshot API and MCP server for developers. It accepts cookie or consent banners like a visitor and removes more than 60 known consent platforms, newsletter popups, and chat widgets before capture; each step can be turned off. Only clean shots are billed: bot checks and CAPTCHAs, blank pages, timeouts, failed loads, and cache hits cost nothing, with the result identified in response headers. Its MCP server offers take_screenshot, get_page_info, and capture_pdf for Claude, Cursor, and other MCP clients.
One GET request returns a PNG, JPEG, WebP, or PDF. Example cURL request:
curl -G "https://api.screenshotneo.com/v1/shot" -d access_key=YOUR_API_KEY --data-urlencode url=https://stripe.com -o shot.webpSee the ScreenshotNeo API documentation for output and request options. ScreenshotNeo includes 1,000 shots a month free with no card; paid plans start at $5 for 3,000. Sign up for free.
Frequently Asked Questions
Can a proxy guarantee that a blocked website will load?
No. Proxy routing changes IP and connection characteristics, but sites can use protections beyond IP blocking.
Does Apify Unblocker support persistent sessions?
No. The documented Unblocker option does not provide a session parameter.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
Quick wins for a faster PC:
Clear out junk files and repair common Windows errorsFree Scan →Fix the driver behind crashes, sound loss and screen glitchesFind Drivers →Repair Windows errors before they cause bigger problemsFix Now →
