If a website presents a CAPTCHA or blocks your capture, treat it as a signal that the site may not permit that automated access—not as a challenge to defeat. Check for an official API, feed, export, or written permission; read the site’s robots.txt and Content Signals; and use a modest, identifiable capture method only when its purpose is allowed. If the site still challenges or blocks you, stop and ask the owner for an approved route.
Why a site works in your browser but blocks a capture
A normal browser visit and an automated request are not necessarily treated alike. A site may assess automated traffic differently, require a session or JavaScript, apply request limits, or present a CAPTCHA when its systems consider a request risky. Google Cloud describes automated scraping as a threat that can collect website data or artifacts. A CAPTCHA is therefore not evidence that you should try a different fingerprint or solve the challenge through a service; it is a reason to check permission and the site’s access policy.
First work out what you need and why. Is the intended use search indexing, a one-time visual record, monitoring, or providing page content to an AI system? Does the site offer an API, data export, sitemap, or feed for that purpose? Is the content public but subject to limits on automated access or reuse? A public page is not automatically permission for every automated purpose.
Check permission before changing your capture setup
Prefer a supported access path
Look for an official API, feed, export, or documented crawler access. If none fits, contact the site owner and request permission or an API key. Confirm the relevant geography, authentication requirements, allowed purpose, retention period, and reuse rights. These are distinct questions: permission to retrieve a page does not by itself settle how long you may store it or whether you may republish its content.
#1 Best Overall
- Compatible Model(s): Magicmoon brand filter only for 24 inch -diagonally measured - widescreen monitor - aspect ratio 16:9 - filter size: width: 20 15/16", Height: 11 13/16" (531mm x 298mm)
- Superior Privacy: The computer privacy filter makes the screen appear dark when looking at it from an angle (the angle is about 30 to 60 degree), but bright when looking directly at it. To change the privacy level - simply adjust your monitor’s brightness accordingly
- Eye and Screen Protection: Privacy Filter does not only protect your private life but also protects your eyes by blocking 30% of blue light , blocking the harmful blue light between 380 to 495 nm, it filters out the blue light and relieves eye strain
- Perfect For Open Workspaces: Great for maintaining screen privacy in open work spaces
- Includes Two Options: Option 1 uses clear adhesive strips that securely attach to any computer screen. Option 2 (for computer screens with a raised bezel only) uses slide mount tabs that easily stick to the display frame, allowing you to slide the privacy screen filter on and off as needed
Read robots.txt and Content Signals
Retrieve the site’s /robots.txt and record the rules that apply to your user agent: Allow, Disallow, any crawl-delay, and any Content-Signal fields. Robots.txt is a crawler instruction, not an access-control mechanism. Google Search Central says that robots.txt instructions cannot enforce crawler behavior; a disallowed URL can still appear in search results. If material must be private, the site owner should use authentication or password protection rather than relying on robots.txt.
Cloudflare’s Content Signals documentation distinguishes search, ai-input, and ai-train. For example, a policy may allow search while disallowing AI training and setting a reference-only use level. Match the signal to the purpose you actually need; do not treat permission for one category as permission for another. Cloudflare states that when a content signal is no, the corresponding use is not permitted. A compliant managed crawler may reject a request that conflicts with those signals rather than trying to work around them.
Do not treat robots.txt as a legal ruling
Robots rules and Content Signals help identify a site’s stated preferences, but they do not establish legal permission for every use. Terms, contracts, privacy obligations, copyright rules, and jurisdiction can matter independently. If the permitted use or reuse is unclear, narrow the task or ask the owner instead of assuming that a reachable URL is fair game.
Rank #2
![[2 Pack] 24 Inch Computer Privacy Screen Filter for 16:9 Widescreen Monitor](https://m.media-amazon.com/images/I/41SBKw3J4ML._SL160_.jpg)
- 【24 PRIVACY FILTER DIMENSIONS】 Width: 20 15/16" (20.9 inches/532 mm), Height: 11 13/16" (11.8 inches/299 mm) - 16:9 Aspect Ratio. Mamol computer privacy filters are designed to be perfectly compatible with HP, Samsung, Dell, Lenovo, Acer, Asus, LG, ViewSonic and other brands of monitors. Please check the width and height dimensions of your computer screen before ordering. If you have any questions about the dimensions, please contact us.
- 【ENHANCED PRIVACY PROTECTION】Mamol 24 inch computer privacy filter keeps your electronic information confidential, making it excellent for use in high traffic areas. the computer privacy screen 24 inch is designed with advanced microlouver technology to block visibility at around 30 degrees and black out screens completely near 60 degrees.
- 【EYES PROTECTION】 This blackout privacy screen greatly reduces eye strain and minimizes potential hazards to vision. It filters 99.9% of UV rays and suppresses 98% of blue light. As a reversible 24-inch privacy screen filter: The glossy side of the protector provides extra clarity and greater privacy, and the matte side minimizes glare and distracting reflections. Satisfy your different daily uses as needed.
- 【BETTER HD CLARTIY】Mamol 24 inch computer privacy screen Shield adds an extra layer of AR Ultra HD light transmission compared to others. It maintains the high definition of the screen without sacrificing too much screen brightness. It won't reduce the brightness and cause eye fatigue because of the privacy screen installed on the screen.
- 【ANTI SCRATCH & WASHABLE 】Our privacy anti-glare Monitor film has a surface enhancement layer to protect the privacy filter from scratches and fingerprints. It is washable and reusable. Even after prolonged use, you will get a brand new privacy screen for your desktop computer monitor after cleaning. Very Durable!
Choose an allowed capture method
| Situation | Practical path | Boundary to observe |
|---|---|---|
| The site offers an API, feed, or export | Use the documented interface and its authentication and quota rules. | Stay within the documented purpose, rate, retention, and reuse terms. |
| The allowed page is server-rendered | A standard HTTP client may be sufficient. | Identify your client, follow applicable crawler rules, pace requests, and stop on a challenge or prohibition. |
| The allowed page requires JavaScript | Use a normal browser session or a managed crawler that documents its policy enforcement. | Browser rendering does not grant permission or guarantee a challenge will pass. |
| The site blocks the intended access or purpose | Stop, contact the owner, request an approved method, or limit the work to explicitly allowed content. | Do not solve CAPTCHAs automatically, rotate IPs to evade controls, spoof another crawler, or keep retrying an explicit block. |
For managed crawling, verify what the service actually enforces. Cloudflare Browser Run documents a /crawl endpoint that respects robots.txt, rejects incompatible Content Signals with a 400 response, and applies a per-domain rate limit. Its documented default is 0.5 seconds between requests to the same domain when the site has not specified a crawl-delay. That default describes this service, not a universal safe delay for other crawlers or a guarantee that a site will accept your traffic.
Recommended Free Tools
Keep request volume controlled and auditable
Use one stable session and a clear identifying user agent, keep concurrency low, cache responses where appropriate, and avoid fetching resources you do not need. Use conditional requests when the server supports them. For a temporary rate-limit response such as HTTP 429, back off exponentially rather than immediately retrying. An explicit prohibition, CAPTCHA loop, or account challenge is a stop condition—not a reason to retry with altered identities or infrastructure.
There is no universal safe request rate. Cloudflare’s documented rate-limiting example for a price-lookup operation uses a managed challenge at 10 requests per 2 minutes and a block at 20 requests per 5 minutes. Those are example rules for that operation, not a general permission threshold. Record the destination, timestamp, user agent, purpose, robots.txt snapshot, response status, and stop condition so the capture can be reviewed and repeated within the same approved scope.
Rank #3
- 【Privacy Filter Dimensions】- Width: 20 15/16" (532 mm), Height: 11 13/16" (299 mm), Diagonal: 24" (609.6 mm) - SightPro Blackout Privacy Screen Filter is engineered to be compatible with HP, Dell, Samsung, Lenovo, LG, Acer, ASUS, ViewSonic, and other monitor brands. Please verify your computer screen's width and height measurements before ordering. It's not recommended to make your selection based solely on your computer screen's diagonal size.
- 【Two Attachment Options】- Installs in minutes. Option 1 uses clear adhesive strips that securely attach to any computer screen. Option 2 (for computer screens with a raised bezel only) uses slide mount tabs that easily stick to the display frame, allowing you to slide the privacy screen filter on and off as needed.
- 【Superior Privacy and Anti Glare】- Our advanced multi-layered film filter blacks out your computer screen when viewing from the side, while maintaining a crystal clear screen straight-on. It also protects your eyes from harmful glare, UV, and blue light. [Note: It does not block visibility directly behind you, regardless of the distance.]
- 【Perfect for Travel and Open Workspaces】- Our computer screen privacy filter is the ideal solution for healthcare providers, mobile workers, commuters, students, and business travelers. Now you can stay compliant and safeguard sensitive corporate information while working in airplanes, subways, airports and public areas.
- 【Package Contents】- Each package includes one privacy screen shield filter, two sets of clear adhesive strips, two sets of slide mount tabs, and a microfiber cleaning cloth. Buy with confidence – located in the US, Sight Pro specializes in providing best-in-class privacy solutions to individuals, small businesses, corporations, government, and educational institutions. Our privacy screens are Section 889 and TAA compliant.
A cautious DIY check with Python
This small script fetches robots.txt, checks whether the configured user agent may fetch one URL, waits for the longer of the configured pause or the published crawl delay, then makes one request. It does not solve a CAPTCHA, retry a rejection, or interpret a Content Signal as permission. Review Content Signals, terms, and the intended reuse separately before running it; if robots.txt cannot be fetched, it fails closed rather than proceeding.
from urllib.error import HTTPError, URLError
from urllib.parse import urlsplit, urlunsplit
from urllib.robotparser import RobotFileParser
from urllib.request import Request, urlopen
import time
USER_AGENT = "ExampleCaptureBot/1.0 (contact: you@example.com)"
TARGET = "https://example.com/page"
MIN_PAUSE_SECONDS = 2.0
def robots_url_for(target):
parts = urlsplit(target)
return urlunsplit((parts.scheme, parts.netloc, "/robots.txt", "", ""))
robots_url = robots_url_for(TARGET)
robots = RobotFileParser()
robots.set_url(robots_url)
try:
# Fetch explicitly so a failed robots.txt request is not silently ignored.
request = Request(robots_url, headers={"User-Agent": USER_AGENT})
with urlopen(request, timeout=20) as response:
if response.status != 200:
raise RuntimeError(f"robots.txt returned HTTP {response.status}; stopping")
robots.parse(response.read().decode("utf-8", errors="replace").splitlines())
except (HTTPError, URLError, TimeoutError, RuntimeError) as exc:
raise SystemExit(f"Could not verify robots.txt; stopping: {exc}")
if not robots.can_fetch(USER_AGENT, TARGET):
raise SystemExit("robots.txt disallows this URL for this user agent; stopping")
crawl_delay = robots.crawl_delay(USER_AGENT) or robots.crawl_delay("*") or 0
# A crawl-delay is a minimum interval; the local pause is deliberately conservative.
time.sleep(max(MIN_PAUSE_SECONDS, crawl_delay))
request = Request(TARGET, headers={"User-Agent": USER_AGENT})
try:
with urlopen(request, timeout=30) as response:
print("HTTP", response.status, "content type:", response.headers.get("Content-Type"))
body = response.read()
if b"captcha" in body.lower():
raise SystemExit("Possible CAPTCHA detected; do not automate a solution. Stop and ask the owner.")
with open("page.bin", "wb") as output:
output.write(body)
except HTTPError as exc:
if exc.code in (403, 429):
raise SystemExit(f"HTTP {exc.code}; stop this capture and resolve access with the owner")
raise SystemExit(f"HTTP {exc.code}; inspect the response and permitted access path")
except (URLError, TimeoutError) as exc:
raise SystemExit(f"Network error: {exc}")Replace the example URL and contact identity with your own. The script checks a single URL and saves the response body; it is not a crawler, a Content Signals parser, or a permission checker. It also cannot reliably identify every CAPTCHA from response text. Inspect the response and browser result yourself, and stop if a challenge, login requirement, or explicit block appears. For a site that requires client-side rendering, use an approved browser or managed crawl path only after the site permits the purpose.
What’s actually slowing this PC down?
Pick the symptom - the matching free tool is one click away.
What to do when a capture fails
- CAPTCHA appears or repeats: stop automated attempts. Request an API, export, or written permission; do not automate solving or send the challenge to a solver.
- HTTP 403 or an explicit block page: treat it as a refusal unless the owner or documented policy explains an authorized remedy. Do not switch IPs, spoof a search crawler, or disguise browser automation.
- HTTP 429: stop the current burst and apply backoff only if the site’s policy allows continued access. Reduce concurrency and request frequency; if the limit remains unclear, ask the owner.
- Blank result or timeout: distinguish a technical load failure from an access decision. Check connectivity and whether the approved page needs JavaScript, but do not repeatedly hammer the endpoint. A managed browser may help with rendering; it does not override access policy.
- robots.txt disallows the path or a signal conflicts with the purpose: do not proceed through that route. Narrow the requested content or request permission.
- Content is private or account-gated: use only credentials and access explicitly authorized for the intended task. Robots.txt is not a substitute for access controls.
Or skip the browser setup: ScreenshotNeo
If you are authorized to capture a page and want a screenshot API rather than maintaining browser infrastructure, ScreenshotNeo can return a PNG, JPEG, WebP, or PDF from one GET request. It accepts consent banners like a visitor and removes more than 60 known consent platforms, newsletter popups, and chat widgets before capture; each of those steps can be turned off. It does not bypass CAPTCHAs: bot checks, blank pages, timeouts, failed loads, and cache hits are not billed, and the response includes X-Page-Verdict and X-Billed headers.
That makes it a capture option, not a way to override a site’s access policy. Confirm the purpose is allowed first; if the page presents a CAPTCHA or explicit block, do not try to defeat it. ScreenshotNeo also provides an MCP server for AI agents with take_screenshot, get_page_info, and capture_pdf tools. The free plan includes 1,000 screenshots per month with no card; paid plans start at $5 for 3,000 shots. All features are on every plan. Yearly billing gives two months free.
Rank #4
![[2-Pack] 24 Inch Computer Privacy Screen Filter for 16:9 Widescreen Monitor](https://m.media-amazon.com/images/I/41jLN+QeBGL._SL160_.jpg)
- 【Improved Privacy Filter】Protescreen 24 inch privacy screen filter after 200 times updates,Use revolutionary micro-louver technology. The 24 inch computer privacy filter limits viewing angle to +/- 28° and provide clear vision on the front. If see from the sides, the greater the angle the darker the screen.Anyone who tries to peek over the side will only see a dark screen! So with a computer privacy screen protector 24 inch, the privacy of your computer screen will never be leaked.
- 【Package Content】You can get 2pcs 24 inch computer monitor privacy screen filter for a better price! Each package includes 24 inch privacy screen film x2, adhesive strips x2, slide mount tabs x2, alcohol x2, cleaning cloth x2. We are a factory that integrates production, processing and sales, We guarantee that all of our products are premium privacy screen protector. If anything happens, we will send you a new 24 inch monitor privacy screen at absolutely no cost. So you can buy with confidence!
- 【Eyes Protection & Anti scratch】Computer screen privacy shield 24 inch monitor use filtering optical materials imported from Japan can reduce 92% of blue light and 98% of UV light, and filter all harmful light emitted from the screen.The high-transparent and reinforced built-in protective layer not only presents high-definition picture quality, but also protects your screen from scratches.Hurry up and place an order, Own privacy screen for computer monitor 24 inch, Protect your screen and eyes.
- 【Brilliant Anti-glare & Function Options】Our privacy screen protector for computer 24 inch monitor protects your eyes by blocking 95% of reflected light. Create a clear and transparent visual space and reduce eye damage by glare. And It is a reversible privacy screen filter. A matte surface effectively prevents blue light and glare, while a glossy is more privacy-resistant. You can choose flexibly according to your needs. In addition to this it also protects your screen from dust and scratches.
- 【Easy to Install & Reusable】Our 24 inch privacy screen for monitor has 2 uniquely designed installation methods: ① Permanent installation- double sided adhesive tape. Suitable for all computers with a screen aspect ratio of 16:9 and a size of 24 inches. ② Removable installation- slide mount tab. Suitable for computer with raised frame, you can slide the filter in and out of the screen as needed, it provide a quick and easy way to remove your monitor privacy filter when you don't need.
Example cURL request (replace the target URL and use your API key):
curl -G "https://api.screenshotneo.com/v1/shot" -d access_key=YOUR_API_KEY --data-urlencode url=https://stripe.com -o shot.webpAPI details: ScreenshotNeo documentation.
Python:
import requests
r = requests.get("https://api.screenshotneo.com/v1/shot", params={"access_key": "YOUR_API_KEY", "url": "https://stripe.com"}, timeout=90)
open("shot.webp", "wb").write(r.content)Node.js:
const q = new URLSearchParams({ access_key: 'YOUR_API_KEY', url: 'https://stripe.com' });
const res = await fetch(`https://api.screenshotneo.com/v1/shot?${q}`);Start with ScreenshotNeo’s free sign-up: 1,000 screenshots a month, no card required.
Free tools Windows power users keep installed
One-click scans. No signup required.
FAQ
Does an Allow rule in robots.txt mean I have permission to reuse a page?
No. It indicates crawler access guidance for that path, not a blanket grant for every purpose, storage period, or reuse. Check the site’s terms and applicable permissions separately.
Best Value
- Compatible Models: Width: 13 9/16" (13.5 inch/344 mm), Height: 7 5/8" (7.6 inch/194 mm), Diagonal: 15.6" (396.24 mm) widescreen laptops which have a 16:9 aspect ratio. Not touchscreen compatible !!! Not fit for 16:10.Do NOT rely solely on your laptop’s diagonal size when ordering. Use a ruler to measure your screen’s visible area (excluding the black bezels). If the width reads 344mm and height reads 194mm, this filter is a perfect match for your device.
- Keep Information Privacy: Effective "black out" privacy from side views outside the 60-degree viewing angle. Designed for optical clarity when viewing from the front, a person not at the front of the screen can only see the dark side of the screen, so it protects buisness secrets and personal privacy
- Eye and Screen Protection: Privacy filter does not only protect your private life but also protects your eyes by blocking 30% of blue light , blocking the harmful blue light between 380 - 495nm, it filters out the blue light and relieves eye strain. Our laptop privacy screen also helps keep your screen safe from dust and scratches
- Perfect For Open Workspaces: Great for maintaining screen privacy in high traffic areas such as open work spaces, airports, airplanes, commuter trains, coffee shops and other public places, etc
- Easy Installation: Choose between 2 simple Options; Slide-On/Off or Mounted. Not touchscreen compatible
Can I keep crawling if robots.txt allows search but blocks AI training?
Only if the actual use fits what is allowed. Search permission does not imply permission for training or other reuse; respect the applicable Content Signal and any stricter site terms.
Is Cloudflare’s 0.5-second crawl interval a safe rate for my scraper?
No. That is Browser Run’s documented default when a site has not set a crawl delay. Other endpoints and site policies can impose different limits.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
