Free tools Windows power users keep installed
One-click scans. No signup required.
Use FormRequest to submit encoded form fields, FormRequest.from_response() when the form is present in a downloaded page, Scrapy’s default cookie middleware to preserve login sessions, and HttpAuthMiddleware for HTTP Basic authentication. These mechanisms solve different problems: a form login is an application-level exchange that commonly creates a cookie session, while Basic authentication sends credentials as part of HTTP authentication. If a page fills itself with JavaScript, find the browser’s underlying request and reproduce that request instead of guessing at the HTML.
Choose the mechanism that matches the site
| Situation | Scrapy approach | Verify |
|---|---|---|
| Known form endpoint and fields | FormRequest |
Action URL, field names, method, encoding and response outcome |
| Form is in a downloaded response | FormRequest.from_response |
Correct form, hidden fields, tokens and submit control |
| Cookie-backed login session | Default CookiesMiddleware |
Later requests use the same session |
| HTTP Basic challenge | HttpAuthMiddleware or request metadata |
Credentials are limited to the protected domain |
| Data appears after browser JavaScript | Reproduce the observed network request | Method, URL, body, headers, tokens and access permission |
Submit a known form with FormRequest
FormRequest URL-encodes the supplied formdata. With no method specified, the request is a POST and the encoded fields go in the body. Set method="GET" when the fields belong in the query string, such as a search form.
import scrapy
class SearchSpider(scrapy.Spider):
name = "search_example"
def start_requests(self):
yield scrapy.FormRequest(
"https://example.org/search",
method="GET",
formdata={"q": "scrapy"},
callback=self.parse_results,
)
def parse_results(self, response):
for item in response.css("article.result"):
yield {
"title": item.css("h2::text").get(),
"url": item.css("a::attr(href)").get(),
}POST versus GET
- Use POST when the endpoint expects submitted form data or changes server-side state.
- Use GET when the site documents a search or filter endpoint whose parameters are intended to appear in the URL.
- Do not infer the endpoint from a visible button alone. Check the form’s
action, method and field names in the response or browser network panel.
Submit a form already present in a response
When a page contains the login form, FormRequest.from_response() can carry forward the form’s submitted controls, including hidden session fields and authentication or CSRF tokens. Override only values that must change, normally the username and password. If several forms exist, select the intended one; if behavior depends on a submit button, identify that control as well.
import scrapy
class LoginSpider(scrapy.Spider):
name = "example_login"
def start_requests(self):
yield scrapy.Request(
"https://example.org/login",
callback=self.parse_login,
)
def parse_login(self, response):
yield scrapy.FormRequest.from_response(
response,
formdata={
"username": "USER_FROM_SECURE_CONFIG",
"password": "SECRET_FROM_SECURE_CONFIG",
},
callback=self.after_login,
)
def after_login(self, response):
# Replace this with a site-specific success check.
if response.css("a[href*='logout']"):
yield scrapy.Request(
"https://example.org/account",
callback=self.parse_account,
)
else:
self.logger.error("Login did not show the expected success marker")
def parse_account(self, response):
yield {"account_title": response.css("title::text").get()}Protect credentials
Keep real credentials in deployment configuration or another secured runtime source, not in a committed spider. Avoid logging passwords, authorization headers or full request bodies. The placeholder names in the example are intentional.
Recommended Free Tools
#1 Best Overall
- Read Before You Buy — No Video Output: These adapters support charging and USB 2.0 data transfer, but cannot transmit video signals. Except for standard USB webcams (which use USB data only), they are not compatible with HDMI/DisplayPort cables, video-capable USB-C hubs, or docking stations with video output.
- Convert USB-A Ports to USB-C: Designed to connect USB-C earphones, cables, flash drives, card readers, and other USB-C accessories to standard USB-A ports. Plug-and-play with no drivers or software required.
- Aluminum Alloy Housing: Built with a sturdy aluminum alloy shell that aids in heat dissipation and protects against daily wear and scratches. Designed to maintain a stable and secure connection.
- Compact & Travel-Friendly: The ultra-compact design allows the adapter to stay plugged into your device without blocking adjacent ports or adding bulk, reducing wear and tear on your original USB ports.
- 12-Month Warranty: Backed by a 12-month manufacturer warranty for peace of mind. Designed to meet strict quality control standards for reliable everyday performance.
Keep login state with CookiesMiddleware
Scrapy’s CookiesMiddleware is enabled by default. It stores cookies received from a site and sends them on later requests, providing the session continuity a browser normally supplies after a successful form login. You generally do not need to copy a session cookie yourself.
yield scrapy.Request(
"https://example.org/account/profile",
callback=self.parse_profile,
)To send a specific cookie, use the request’s cookies argument:
yield scrapy.Request(
"https://example.org/private",
cookies={"feature_flag": "enabled"},
callback=self.parse_private,
)Do not set a raw Cookie header and expect it to be managed by the middleware; the middleware drops manually supplied Cookie headers. Disable cookie handling only when you deliberately want stateless requests with COOKIES_ENABLED = False.
Inspect cookies while debugging
Set COOKIES_DEBUG = True to log cookies sent and received. Treat those logs as sensitive: a session cookie may grant account access, so restrict log access and remove verbose cookie logging after diagnosis.
Rank #2
- 5-in-1 USB-C Hub: Experience comprehensive connectivity featuring a Power Delivery input, two USB-A 2.0 ports, a USB-A 3.0 port, and an HDMI port. (Note: The USB-C power delivery input port is only for connecting an external wall charger to power your laptop and cannot power peripheral devices.)
- 90W Pass-Through Charging: Achieve optimal charging with 90W pass-through power to your laptop, supported by a total input of 100W, with the hub reserving 10W for operational efficiency. (Note: Wall charger not included.)
- Quick Data Transfers: Accelerate your productivity with rapid data transfers using a high-speed 5Gbps USB 3.0 port and two 480Mbps USB 2.0 ports.
- 4K HDMI Display: Enhance your visual experience with a hub capable of delivering 4K resolution at 30Hz in both mirror and extend modes. Please note that this hub is compatible with MacBook (macOS 12 and newer), Windows 10 and 11, ChromeOS, and laptops equipped with DP Alt Mode and Power Delivery. Note: This device is not compatible with Linux.
- What You Get: Anker USB-C Hub (5-in-1, 4K HDMI), welcome guide, 18-month warranty, and our friendly customer service.
Use HTTP Basic authentication correctly
Scrapy’s HttpAuthMiddleware “authenticates requests using Basic access authentication (aka. HTTP auth).” It is for an HTTP Basic challenge, not for filling a website’s HTML login form.
Spider-wide settings
# settings.py
HTTPAUTH_USER = "user-from-secure-config"
HTTPAUTH_PASS = "password-from-secure-config"
HTTPAUTH_DOMAIN = "protected.example.org"These settings suit credentials that remain stable for the spider run.
Per-request credentials
yield scrapy.Request(
"https://protected.example.org/report",
meta={
"http_user": "USER_FROM_SECURE_CONFIG",
"http_pass": "SECRET_FROM_SECURE_CONFIG",
"http_auth_domain": "protected.example.org",
},
callback=self.parse_report,
)Always set the domain. If the domain is left unset (or None), credentials can be sent to every request, including unrelated hosts in a multi-domain crawl. Limit the scope to the exact protected host or the narrowest domain required.
Referer and cross-scheme leakage
URLs in a Referer can disclose crawled paths to another site. Scrapy’s default policy avoids sending a referrer from HTTPS to HTTP; a stricter policy such as same-origin or no-referrer may be appropriate when URLs contain sensitive information.
Rank #3
- Sleek 7-in-1 USB-C Hub: Features an HDMI port, two USB-A 3.0 ports, and a USB-C data port, each providing 5Gbps transfer speeds. It also includes a USB-C PD input port for charging up to 100W and dual SD and TF card slots, all in a compact design.
- Flawless 4K@60Hz Video with HDMI: Delivers exceptional clarity and smoothness with its 4K@60Hz HDMI port, making it ideal for high-definition presentations and entertainment. (Note: Only the HDMI port supports video projection; the USB-C port is for data transfer only.)
- Double Up on Efficiency: The two USB-A 3.0 ports and a USB-C port support a fast 5Gbps data rate, significantly boosting your transfer speeds and improving productivity.
- Fast and Reliable 85W Charging: Offers high-capacity, speedy charging for laptops up to 85W, so you spend less time tethered to an outlet and more time being productive.
- What You Get: Anker USB-C Hub (7-in-1), welcome guide, 18-month warranty, and our friendly customer service.
When JavaScript performs the real login or data request
A downloaded HTML page may contain no usable form because JavaScript submits an XHR or fetch request. Open browser developer tools, submit the form, and inspect the request that returns the login result or data.
- Record the request method and full URL.
- Capture the body format: URL-encoded fields, JSON, multipart data or no body.
- Note required headers, cookies, authorization values, origin checks and CSRF tokens.
- Check whether a prior request creates a token or cookie that must be replayed.
- Reproduce the smallest equivalent request in Scrapy, then add only the headers and fields the server actually requires.
Most browser developer tools can copy a request as cURL. Scrapy can construct an equivalent Request from a cURL command, which is often faster and less error-prone than manually transcribing a complex request. Reproducing every browser request can require substantial effort; do not assume browser automation is always necessary, but do verify the endpoint’s authorization and access rules.
import json
import scrapy
class ApiLoginSpider(scrapy.Spider):
name = "api_login"
def start_requests(self):
yield scrapy.Request(
"https://example.org/login",
callback=self.parse_login_page,
)
def parse_login_page(self, response):
token = response.css("input[name=csrf_token]::attr(value)").get()
yield scrapy.Request(
"https://example.org/api/session",
method="POST",
headers={"Content-Type": "application/json"},
body=json.dumps({
"username": "USER_FROM_SECURE_CONFIG",
"password": "SECRET_FROM_SECURE_CONFIG",
"csrf_token": token,
}),
callback=self.after_login,
)
def after_login(self, response):
# Confirm the API's documented success signal before crawling onward.
if response.status == 200:
yield scrapy.Request("https://example.org/api/me", callback=self.parse_me)
def parse_me(self, response):
yield json.loads(response.text)Diagnose a login that appears to fail
HTTP 200 but you are still logged out
A 200 status only proves that the server returned a response. Check for a site-specific authenticated marker, the redirect destination, an account-only endpoint or an explicit error message. Then verify the form action, field names, hidden inputs, submit control and cookies.
Missing CSRF or session token
Start with a GET of the form page and use from_response() so hidden fields are preserved. If the token is generated by JavaScript, inspect the request that obtains it and reproduce that prerequisite request.
Quick wins for a faster PC:
Fix the driver behind crashes, sound loss and screen glitchesFind Drivers →Clear out junk files and repair common Windows errorsFree Scan →Scan for outdated or missing drivers - takes under a minuteDriver Scan →Rank #4
- Dual Converters, Infinite Potential：Includes 2× USB C male to USB A female adapters and 2× USB A male to USB C female adapters. Perfect for a wide range of uses—tablets with Bluetooth keyboards, expand USB ports on macbook, and more. Two different converters for all your daily needs
- Next-Level 10Gbps & 3A Charging: No more slow 480Mbps, this usb to usb c adapter has a transfer speed of up to 10Gbps, allowing you to do more transferring in less time. This usb adapter fits both USB A and USB C charger, supporting up to 3A fast charging
- Upgraded Exquisite Craftsmanship: With an aluminum alloy housing and metal connector, the usbc to usb adapter is extremely durable and sturdy. Rigorously tested to withstand more than 10,000 times of plugging and unplugging, ensuring long-lasting performance
- Broad Compatible: The usb c to usb adapter widely supports all USB C/ USB A devices like laptops, tablets, cellphones, car chargers, and phone chargers. Such as compatible with MacBook Pro/Air 2023/2022, Thunderbolt 4/3 Devices,Apple MagSafe Watch 9/8/7/SE/Ultra, iPad Pro 2022/2021, Samsung Galaxy S23/S20/S10, and iPhone 17/16/15 Pro. Plug and play
- Please Note: To reach 10Gbps speed, keep the cable under 3.3 ft. For USB A Male to USB C adapters, try flipping the USB C connector. USB C Male to USB A adapters support bidirectional 10Gbps transfer within 3.3 ft
Cookies are not carried forward
Confirm that cookie handling is enabled, that subsequent requests use the same session context, and that you did not place cookies in a raw Cookie header. Temporarily enable COOKIES_DEBUG in a protected environment.
Basic credentials reach the wrong host
Set HTTPAUTH_DOMAIN or per-request http_auth_domain explicitly. Review redirects and multi-domain start URLs so credentials remain inside the intended boundary.
Form fields look correct but the site rejects the request
Compare the browser request with Scrapy: URL, method, encoding, body, headers, origin or referer checks, cookies and token freshness. A visible form can submit to a different endpoint or use a JSON request behind the scenes.
Redirects change the result
Inspect the final response URL and redirect chain. A successful login may redirect to an account page, while a failed login may return the same form with an error. Base your callback decision on the target site’s actual success signal.
Best Value
- 5-in-1 Connectivity: Equipped with a 4K HDMI port, a 5 Gbps USB-C data port, two 5 Gbps USB-A ports, and a USB C 100W PD-IN port. Note: The USB C 100W PD-IN port supports only charging and does not support data transfer devices such as headphones or speakers.
- Powerful Pass-Through Charging: Supports up to 85W pass-through charging so you can power up your laptop while you use the hub. Note: Pass-through charging requires a charger (not included). Note: To achieve full power for iPad, we recommend using a 45W wall charger.
- Transfer Files in Seconds: Move files to and from your laptop at speeds of up to 5 Gbps via the USB-C and USB-A data ports. Note: The USB C 5Gbps Data port does not support video output.
- HD Display: Connect to the HDMI port to stream or mirror content to an external monitor in resolutions of up to 4K@30Hz. Note: The USB-C ports do not support video output.
- What You Get: Anker 332 USB-C Hub (5-in-1), welcome guide, our worry-free 18-month warranty, and friendly customer service.
Performance, reliability and security notes
- Fetch the login page once per session and reuse the cookie jar rather than logging in before every item.
- Use explicit callbacks that verify authentication before launching a large crawl.
- Keep retries and timeouts appropriate to the target service; repeated credential submissions can trigger lockouts.
- Do not disable TLS verification or weaken authentication checks to make a request “work.”
- Respect the site’s authorization, terms and applicable access rules. Scrapy’s ability to send a request does not grant permission to access the endpoint.
- Separate credentials by environment and rotate them according to your organization’s policy.
Or skip the browser setup
If your goal is a clean image or PDF of a page rather than crawling its authenticated data, ScreenshotNeo provides a single screenshot request. It accepts consent banners before capture and removes more than 60 known consent platforms, newsletter popups and chat widgets; each cleanup step can be disabled. Only clean shots are billed: bot checks or CAPTCHAs, blank pages, timeouts, failed loads and cache hits cost nothing, and response headers identify the page verdict and billing result. Its MCP server gives Claude, Cursor and other MCP clients take_screenshot, get_page_info and capture_pdf tools. The Free plan includes 1,000 shots per month without a card; paid plans start at $5 for 3,000 shots.
See the ScreenshotNeo API documentation for options such as cookies, custom headers, JavaScript, waiting for selectors or network idle, device presets, full-page capture and PDFs.
curl -G "https://api.screenshotneo.com/v1/shot" -d access_key=YOUR_API_KEY --data-urlencode url=https://stripe.com -o shot.webpimport requests
r = requests.get("https://api.screenshotneo.com/v1/shot", params={"access_key": "YOUR_API_KEY", "url": "https://stripe.com"}, timeout=90)
open("shot.webp", "wb").write(r.content)const q = new URLSearchParams({ access_key: 'YOUR_API_KEY', url: 'https://stripe.com' });
const res = await fetch(`https://api.screenshotneo.com/v1/shot?${q}`);Create a free ScreenshotNeo account to get 1,000 screenshots each month with no card.
FAQ
Does Scrapy manage cookies automatically?
Yes. CookiesMiddleware is enabled by default and retains cookies received from a site for later requests, unless you disable it or deliberately use a separate cookie policy.
Windows Errors? Fix Them Before They Spread
Repair common Windows errors and clear accumulated junk for a smoother, more stable PC - no reinstall needed.Free scan · no reinstallOutdated Drivers Are Slowing You Down
One free scan finds every outdated or missing driver and matches the right update for your exact hardware.Free scan · exact hardware matchHow can I see cookies being sent and received?
Enable COOKIES_DEBUG = True while diagnosing the session, then disable it or restrict log access because session cookies are sensitive.
Can Basic authentication replace a login form?
No. Basic authentication and an application login form are different protocols. Use the mechanism the server actually requires.
Which Scrapy version should I follow?
Check the documentation and installed package version together. Stable documentation identifies Scrapy 2.19.0, while some detailed pages are served from a master branch that may include unreleased changes; verify helper names and signatures locally before deployment.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
