Free tools Windows power users keep installed
One-click scans. No signup required.
Use Crawlee when you need a maintained scraping framework rather than a one-off HTTP request. This tutorial builds a small JavaScript crawler, saves structured records, follows links safely, and then shows when to switch from CheerioCrawler to PlaywrightCrawler or PuppeteerCrawler. Crawlee is an open-source web-scraping library for JavaScript and Python. The current JavaScript quick start is labeled version 3.18 and requires Node.js 16 or later; confirm version-sensitive details in the documentation before deploying.
What you will build
The first example uses CheerioCrawler. It requests HTML directly, extracts a page title and description, follows links, and writes JSON records to Crawlee’s default dataset. This is the right starting point when the data is present in the server-delivered HTML and you do not need clicks, scrolling, or JavaScript execution.
- JavaScript project created with the Crawlee CLI
- A maximum of 50 requests while learning
- Title, description, and URL fields in a dataset
- Link discovery through Crawlee’s request queue
Use only URLs you are permitted to access. Check a site’s terms, robots guidance, privacy obligations, and rate limits before running a crawl.
Install Crawlee and create a project
- Install Node.js 16 or newer.
- Run the official project generator:
npx crawlee create my-crawler cd my-crawler - Start the generated project once to verify the installation:
npm start
The generator creates a module-enabled JavaScript project. If you prefer to add Crawlee to an existing project, install it with:
#1 Best Overall
- Read Before You Buy — No Video Output: These adapters support charging and USB 2.0 data transfer, but cannot transmit video signals. Except for standard USB webcams (which use USB data only), they are not compatible with HDMI/DisplayPort cables, video-capable USB-C hubs, or docking stations with video output.
- Convert USB-A Ports to USB-C: Designed to connect USB-C earphones, cables, flash drives, card readers, and other USB-C accessories to standard USB-A ports. Plug-and-play with no drivers or software required.
- Aluminum Alloy Housing: Built with a sturdy aluminum alloy shell that aids in heat dissipation and protects against daily wear and scratches. Designed to maintain a stable and secure connection.
- Compact & Travel-Friendly: The ultra-compact design allows the adapter to stay plugged into your device without blocking adjacent ports or adding bulk, reducing wear and tear on your original USB ports.
- 12-Month Warranty: Backed by a 12-month manufacturer warranty for peace of mind. Designed to meet strict quality control standards for reliable everyday performance.
npm install crawleeCheerioCrawler is included in the Crawlee package. Browser crawlers require a separate browser dependency, such as playwright or puppeteer.
Write a first CheerioCrawler
Replace the generated entry file with this example. Change startUrls to a site you are allowed to crawl. The selectors are deliberately generic; inspect your target site’s HTML and adjust them rather than assuming every site uses the same markup.
import { CheerioCrawler, Dataset } from 'crawlee';
const crawler = new CheerioCrawler({
maxRequestsPerCrawl: 50,
async requestHandler({ request, $, enqueueLinks, log }) {
const title = $('title').first().text().trim();
const description = $('meta[name="description"]').attr('content')?.trim() ?? null;
await Dataset.pushData({
url: request.loadedUrl ?? request.url,
title,
description,
});
await enqueueLinks({
selector: 'a[href]',
label: 'detail',
});
log.info(`Saved ${request.url}`);
},
});
await crawler.run([
'https://example.com/',
]);The handler receives a request and a Cheerio document-like $ function. Dataset.pushData() appends one JSON object per page. enqueueLinks() adds discovered links to Crawlee’s request queue, so the crawler can continue beyond the starting page. The 50-request cap prevents an accidental crawl from expanding indefinitely while you learn.
Restrict which links are followed
For a real site, constrain discovery to the host and paths you need. For example:
The Tool Desk
Outbyte PC Repair FREEClear out junk files and repair common Windows errorsFree Scan →Outbyte Driver Updater FREEFix the driver behind crashes, sound loss and screen glitchesFind Drivers →await enqueueLinks({
globs: ['https://example.com/products/**'],
selector: 'a[href]',
});You can also enqueue a known set of URLs directly with await crawler.addRequests([...]) before running, or use labels to route different page types to different handlers. Keep the request limit and filtering rules explicit in source control.
Rank #2
- 5-in-1 USB-C Hub: Experience comprehensive connectivity featuring a Power Delivery input, two USB-A 2.0 ports, a USB-A 3.0 port, and an HDMI port. (Note: The USB-C power delivery input port is only for connecting an external wall charger to power your laptop and cannot power peripheral devices.)
- 90W Pass-Through Charging: Achieve optimal charging with 90W pass-through power to your laptop, supported by a total input of 100W, with the hub reserving 10W for operational efficiency. (Note: Wall charger not included.)
- Quick Data Transfers: Accelerate your productivity with rapid data transfers using a high-speed 5Gbps USB 3.0 port and two 480Mbps USB 2.0 ports.
- 4K HDMI Display: Enhance your visual experience with a hub capable of delivering 4K resolution at 30Hz in both mirror and extend modes. Please note that this hub is compatible with MacBook (macOS 12 and newer), Windows 10 and 11, ChromeOS, and laptops equipped with DP Alt Mode and Power Delivery. Note: This device is not compatible with Linux.
- What You Get: Anker USB-C Hub (5-in-1, 4K HDMI), welcome guide, 18-month warranty, and our friendly customer service.
Find your saved data
Local runs use a storage directory in the current working directory. The default dataset is written as JSON files under ./storage/datasets/default/. Open those files to inspect records, load them into a database, or transform them into CSV in a later pipeline.
To place Crawlee’s local storage elsewhere, set CRAWLEE_STORAGE_DIR before starting the program:
CRAWLEE_STORAGE_DIR=./crawler-data npm startOn Windows PowerShell, use $env:CRAWLEE_STORAGE_DIR=".crawler-data"; npm start. Keep storage outside a temporary directory when you need to resume or audit a run.
Windows Errors? Fix Them Before They Spread
Repair common Windows errors and clear accumulated junk for a smoother, more stable PC - no reinstall needed.Free scan · no reinstallOutdated Drivers Are Slowing You Down
One free scan finds every outdated or missing driver and matches the right update for your exact hardware.Free scan · exact hardware matchChoose the right crawler
| Requirement | Starting point | Trade-off |
|---|---|---|
| Content is in ordinary HTML and speed or low setup matters | CheerioCrawler |
Fast plain-HTTP fetching and HTML parsing, but it does not execute page JavaScript. |
| JavaScript rendering, clicks, scrolling, or browser state is required | PlaywrightCrawler |
More capable browser automation, with a separate Playwright install and browser runtime. |
| Your team already uses Puppeteer | PuppeteerCrawler |
Supported browser automation, with Puppeteer installed separately. |
The crawler classes share a common interface, so the request queue, datasets, and many lifecycle concepts transfer between them. Migration is not automatic: browser-specific selectors, waits, page interactions, and handler arguments still need review.
Use Playwright for rendered pages
Install the browser dependency in the project:
npm install crawlee playwrightA minimal browser crawler looks like this:
import { PlaywrightCrawler, Dataset } from 'crawlee';
const crawler = new PlaywrightCrawler({
maxRequestsPerCrawl: 20,
headless: true,
async requestHandler({ page, request, enqueueLinks }) {
await page.waitForLoadState('domcontentloaded');
const title = await page.title();
const heading = await page.locator('h1').first().textContent().catch(() => null);
await Dataset.pushData({
url: request.loadedUrl ?? request.url,
title,
heading: heading?.trim() ?? null,
});
await enqueueLinks({ selector: 'a[href]' });
},
});
await crawler.run(['https://example.com/']);Set headless: false while developing if you need to watch the browser. A visible browser is useful for diagnosing consent dialogs, navigation, and selectors; use headless mode for unattended runs.
Rank #3
- Sleek 7-in-1 USB-C Hub: Features an HDMI port, two USB-A 3.0 ports, and a USB-C data port, each providing 5Gbps transfer speeds. It also includes a USB-C PD input port for charging up to 100W and dual SD and TF card slots, all in a compact design.
- Flawless 4K@60Hz Video with HDMI: Delivers exceptional clarity and smoothness with its 4K@60Hz HDMI port, making it ideal for high-definition presentations and entertainment. (Note: Only the HDMI port supports video projection; the USB-C port is for data transfer only.)
- Double Up on Efficiency: The two USB-A 3.0 ports and a USB-C port support a fast 5Gbps data rate, significantly boosting your transfer speeds and improving productivity.
- Fast and Reliable 85W Charging: Offers high-capacity, speedy charging for laptops up to 85W, so you spend less time tethered to an outlet and more time being productive.
- What You Get: Anker USB-C Hub (7-in-1), welcome guide, 18-month warranty, and our friendly customer service.
When Puppeteer is the better choice
Choose PuppeteerCrawler when an existing codebase, team skill set, or dependency is already centered on Puppeteer. Install Puppeteer separately and keep browser-specific code isolated so a later change of crawler class does not require rewriting data modeling and storage.
Python support
Crawlee also supports Python. Do not mix the JavaScript installation commands with the Python setup. The Python quick start uses an asynchronous entry point and a PlaywrightCrawler; install the Python package and browser tooling according to the current Python documentation, then adapt this pattern:
Do these 3 things before closing this tab:
1Clear out junk files and repair common Windows errors2Scan for outdated or missing drivers - takes under a minute3Repair Windows errors before they cause bigger problemsimport asyncio
from crawlee.crawlers import PlaywrightCrawler
async def main():
crawler = PlaywrightCrawler(max_requests_per_crawl=20)
@crawler.router.default_handler
async def handler(context):
title = await context.page.title()
await context.push_data({
"url": context.request.url,
"title": title,
})
await crawler.run(["https://example.com/"])
if __name__ == "__main__":
asyncio.run(main())Python runs use the same default local dataset location, ./storage/datasets/default/, unless you configure another storage directory. The exact import paths and browser-install commands are version-sensitive, so check the Python quick start for the release you install.
Production controls that matter
Sessions and cookies
Crawlee sessions keep identity-bound state such as cookies together across requests. This is useful for a site that expects a consistent session, but it does not grant permission, anonymity, or guaranteed access.
Proxies
ProxyConfiguration can select proxy URLs and associate a stable proxy URL with a supplied session ID. Treat this as request-management infrastructure, not a promise that rotation prevents blocking. Use proxies only lawfully and respect the target’s policies.
Rank #4
- Dual Converters, Infinite Potential：Includes 2× USB C male to USB A female adapters and 2× USB A male to USB C female adapters. Perfect for a wide range of uses—tablets with Bluetooth keyboards, expand USB ports on macbook, and more. Two different converters for all your daily needs
- Next-Level 10Gbps & 3A Charging: No more slow 480Mbps, this usb to usb c adapter has a transfer speed of up to 10Gbps, allowing you to do more transferring in less time. This usb adapter fits both USB A and USB C charger, supporting up to 3A fast charging
- Upgraded Exquisite Craftsmanship: With an aluminum alloy housing and metal connector, the usbc to usb adapter is extremely durable and sturdy. Rigorously tested to withstand more than 10,000 times of plugging and unplugging, ensuring long-lasting performance
- Broad Compatible: The usb c to usb adapter widely supports all USB C/ USB A devices like laptops, tablets, cellphones, car chargers, and phone chargers. Such as compatible with MacBook Pro/Air 2023/2022, Thunderbolt 4/3 Devices,Apple MagSafe Watch 9/8/7/SE/Ultra, iPad Pro 2022/2021, Samsung Galaxy S23/S20/S10, and iPhone 17/16/15 Pro. Plug and play
- Please Note: To reach 10Gbps speed, keep the cable under 3.3 ft. For USB A Male to USB C adapters, try flipping the USB C connector. USB C Male to USB A adapters support bidirectional 10Gbps transfer within 3.3 ft
Concurrency and limits
Start with a small maxRequestsPerCrawl and conservative concurrency. Increase gradually after measuring response errors, memory use, and the site’s stated limits. Browser crawlers consume substantially more resources than direct HTTP crawlers because each page needs browser processes and rendering.
Recommended Free Tools
Rendering and data quality
Prefer CheerioCrawler when its HTML contains the fields you need. Use Playwright only for the pages that require rendering or interaction. Save the source URL, a crawl timestamp, and a stable identifier when your downstream process needs deduplication or auditing.
Independent reader supportYour contribution helps us test, update, and keep practical guides available for everyone.Troubleshooting
“Cannot find module” or browser launch errors
Install the package that matches your crawler: npm install crawlee playwright for Playwright or the equivalent Puppeteer package. Browser libraries may also require a separate browser download; follow the installed version’s setup output.
Fields are empty in CheerioCrawler
Inspect the raw HTML returned over HTTP. If the content appears only after JavaScript runs, Cheerio cannot see it; switch that route to PlaywrightCrawler and wait for the relevant selector.
The crawler stops after a few pages
Check maxRequestsPerCrawl, URL filters, and whether links are being enqueued. A 50-request cap in a learning example is intentional. Review logs for HTTP errors and duplicate URLs.
Best Value
- 5-in-1 Connectivity: Equipped with a 4K HDMI port, a 5 Gbps USB-C data port, two 5 Gbps USB-A ports, and a USB C 100W PD-IN port. Note: The USB C 100W PD-IN port supports only charging and does not support data transfer devices such as headphones or speakers.
- Powerful Pass-Through Charging: Supports up to 85W pass-through charging so you can power up your laptop while you use the hub. Note: Pass-through charging requires a charger (not included). Note: To achieve full power for iPad, we recommend using a 45W wall charger.
- Transfer Files in Seconds: Move files to and from your laptop at speeds of up to 5 Gbps via the USB-C and USB-A data ports. Note: The USB C 5Gbps Data port does not support video output.
- HD Display: Connect to the HDMI port to stream or mirror content to an external monitor in resolutions of up to 4K@30Hz. Note: The USB-C ports do not support video output.
- What You Get: Anker 332 USB-C Hub (5-in-1), welcome guide, our worry-free 18-month warranty, and friendly customer service.
A selector works manually but not in automation
Wait for the element in a browser crawler, verify the selector against the current DOM, and account for frames, consent dialogs, or login state. Do not hide a failed extraction by writing empty records without logging it.
Data files are missing
Look in the process working directory, not necessarily the source-file directory, and check CRAWLEE_STORAGE_DIR. A successful run should create JSON files under storage/datasets/default/.
Or skip the browser setup
For a single clean screenshot or an automated visual capture, ScreenshotNeo provides a GET API and an MCP server for AI agents. It accepts consent banners before capture and removes more than 60 known consent platforms, newsletter popups, and chat widgets; each cleanup step can be disabled. Only clean shots are billed: bot checks or CAPTCHAs, blank pages, timeouts, failed loads, and cache hits are not billed, and response headers identify the page verdict and billing result.
One-call cURL example (see the ScreenshotNeo documentation):
curl -G "https://api.screenshotneo.com/v1/shot" -d access_key=YOUR_API_KEY --data-urlencode url=https://stripe.com -o shot.webpPython:
import requests
r = requests.get("https://api.screenshotneo.com/v1/shot", params={"access_key": "YOUR_API_KEY", "url": "https://stripe.com"}, timeout=90)
open("shot.webp", "wb").write(r.content)Node.js:
const q = new URLSearchParams({ access_key: 'YOUR_API_KEY', url: 'https://stripe.com' });
const res = await fetch(`https://api.screenshotneo.com/v1/shot?${q}`);ScreenshotNeo supports PNG, JPEG, WebP, and PDF output, browser/device settings, custom CSS and JavaScript, waits, selectors, blocking rules, headers, cookies, geolocation, caching, signed links, asynchronous jobs, webhooks, bulk capture, and an MCP server with take_screenshot, get_page_info, and capture_pdf. The free plan includes 1,000 screenshots per month with no card; paid plans start at $5 for 3,000. Create a free ScreenshotNeo account.
Next steps
Once the small crawl is reliable, read Crawlee’s guides for request and result storage, rendering, configuration, proxies, sessions, scaling, Docker, avoiding blocks, and parallel scraping. Add one capability at a time: first validation and retries, then persistence, then concurrency or browser interaction. That progression makes failures attributable and keeps a prototype from becoming an uncontrolled crawl.
Frequently Asked Questions
Can Crawlee scrape pages that require login?
It can automate authenticated sessions when you supply lawful credentials and implement the site’s required login flow, cookies, or headers. Store secrets outside source control and verify that automated access is permitted.
Does Crawlee itself provide proxy IPs?
Crawlee provides proxy configuration and session-management features; you must supply or obtain proxy URLs separately.
Should every crawler use Playwright?
No. Use CheerioCrawler when the needed fields are already in the HTTP response. Browser automation adds setup and resource cost and is justified by rendering or interaction requirements.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
