Do these 3 things before closing this tab:
1Repair Windows errors before they cause bigger problems2Scan for outdated or missing drivers - takes under a minute3Clear out junk files and repair common Windows errorsCrawlee is an open-source library for building web scrapers and browser-automation workflows. Choose CheerioCrawler when the data is available in fetched HTML; choose PlaywrightCrawler or PuppeteerCrawler when the page needs JavaScript execution or browser behavior. The package does not include Playwright or Puppeteer, so install the browser automation dependency separately.
What is Crawlee?
Crawlee is a library for building crawlers that fetch pages, extract data, and, when needed, control a browser. It has JavaScript and Python implementations; the code below uses JavaScript. The project is open source under the Apache License 2.0, according to the repository README.
Crawlee provides common crawler components and a shared style of working across different crawler types. You choose the fetching and rendering approach that suits the site, then implement the page-specific extraction and navigation logic. Crawlee does not automatically understand a site’s data structure or make unreliable selectors correct; the project describes its role as making crawler development and maintenance easier, not fixing broken selectors for you.
Should I use CheerioCrawler or PlaywrightCrawler?
Start by checking where the information appears. If it is present in the HTML returned by an ordinary HTTP request, an HTTP crawler avoids launching a browser. If the page builds its content with JavaScript, or the task depends on browser interactions, use a browser crawler. Crawlee also offers PuppeteerCrawler for teams with a Puppeteer workflow. The official JavaScript quick start describes CheerioCrawler as fast and efficient, but does not establish a general benchmark that applies to every site or workload.
#1 Best Overall
- Read Before You Buy — No Video Output: These adapters support charging and USB 2.0 data transfer, but cannot transmit video signals. Except for standard USB webcams (which use USB data only), they are not compatible with HDMI/DisplayPort cables, video-capable USB-C hubs, or docking stations with video output.
- Convert USB-A Ports to USB-C: Designed to connect USB-C earphones, cables, flash drives, card readers, and other USB-C accessories to standard USB-A ports. Plug-and-play with no drivers or software required.
- Aluminum Alloy Housing: Built with a sturdy aluminum alloy shell that aids in heat dissipation and protects against daily wear and scratches. Designed to maintain a stable and secure connection.
- Compact & Travel-Friendly: The ultra-compact design allows the adapter to stay plugged into your device without blocking adjacent ports or adding bulk, reducing wear and tear on your original USB ports.
- 12-Month Warranty: Backed by a 12-month manufacturer warranty for peace of mind. Designed to meet strict quality control standards for reliable everyday performance.
| Need | Start with | What to know |
|---|---|---|
| Fetch and parse server-rendered or static HTML | CheerioCrawler |
Uses HTTP and HTML parsing; it does not render client-side JavaScript. |
| Render JavaScript or interact with a browser page | PlaywrightCrawler |
Controls a browser through Playwright; install Playwright separately. |
| Continue an existing Puppeteer-based workflow | PuppeteerCrawler |
Provides a Puppeteer browser crawler with Crawlee’s crawler interface; install Puppeteer separately. |
The trade-off is not simply speed. HTTP parsing typically requires fewer browser-specific dependencies and resources, while a browser can execute page scripts and support interactions unavailable to an HTTP parser. Team familiarity also matters: if your project already uses Playwright or Puppeteer, matching that tool may reduce integration work. Neither choice guarantees that a particular site’s content will be accessible or that its terms allow automated collection.
What do I need before installing Crawlee?
- For the JavaScript quick-start path, use Node.js 16 or later. Check the current quick start for changes before setting up a new project.
- Install the Crawlee package with npm. Browser automation libraries are separate dependencies, not bundled with Crawlee.
- Use an explicit request limit in early experiments so a test crawl has a bounded scope.
- Confirm that your collection and automation comply with the target site’s rules and applicable law. Proxy and session features are configuration mechanisms, not permission or a guarantee that restrictions can be bypassed.
The official JavaScript API documentation currently identifies its docs as version 3.18. Its changelog lists version 3.18.1 dated August 12, 2026, and version 3.18.0 dated August 4, 2026. These are documentation and changelog details, not a claim that every installed project is on that version; check the live changelog and your package lockfile when diagnosing version-specific behavior.
How do I scrape a website with Crawlee?
A small crawl has four parts: install the appropriate packages, define what each request should extract, set a limit, and store the results. The examples below use a documentation page as the target; replace it with a site you are permitted to crawl. They are starting points: selectors and extraction fields must match the actual pages you process.
Rank #2
- 5-in-1 USB-C Hub: Experience comprehensive connectivity featuring a Power Delivery input, two USB-A 2.0 ports, a USB-A 3.0 port, and an HDMI port. (Note: The USB-C power delivery input port is only for connecting an external wall charger to power your laptop and cannot power peripheral devices.)
- 90W Pass-Through Charging: Achieve optimal charging with 90W pass-through power to your laptop, supported by a total input of 100W, with the hub reserving 10W for operational efficiency. (Note: Wall charger not included.)
- Quick Data Transfers: Accelerate your productivity with rapid data transfers using a high-speed 5Gbps USB 3.0 port and two 480Mbps USB 2.0 ports.
- 4K HDMI Display: Enhance your visual experience with a hub capable of delivering 4K resolution at 30Hz in both mirror and extend modes. Please note that this hub is compatible with MacBook (macOS 12 and newer), Windows 10 and 11, ChromeOS, and laptops equipped with DP Alt Mode and Power Delivery. Note: This device is not compatible with Linux.
- What You Get: Anker USB-C Hub (5-in-1, 4K HDMI), welcome guide, 18-month warranty, and our friendly customer service.
Option 1: Fetch HTML with CheerioCrawler
In a new Node.js project, install Crawlee:
npm install crawleeSave this as crawler.js and run node crawler.js. It records each requested page’s title and URL in Crawlee’s default dataset and stops after the configured maximum of ten requests.
import { CheerioCrawler, Dataset } from 'crawlee';
const crawler = new CheerioCrawler({
async requestHandler({ request, $, log }) {
const title = $('title').first().text().trim();
await Dataset.pushData({
url: request.url,
title,
});
log.info(`Saved ${request.url}`);
},
});
await crawler.run(['https://crawlee.dev/js/docs/quick-start'], {
maxRequestsPerCrawl: 10,
});This example processes the URL you provide; it does not discover a site’s entire link structure. To follow links, add link discovery and a clear scope appropriate to the site. In a real extraction job, inspect the returned HTML, use selectors that match the target’s markup, handle missing fields, and validate a few saved records before increasing crawl scope. If the useful content is inserted only after JavaScript runs, this crawler will not make it appear.
Option 2: Render pages with PlaywrightCrawler
Install Crawlee and Playwright as separate packages:
Rank #3
- Sleek 7-in-1 USB-C Hub: Features an HDMI port, two USB-A 3.0 ports, and a USB-C data port, each providing 5Gbps transfer speeds. It also includes a USB-C PD input port for charging up to 100W and dual SD and TF card slots, all in a compact design.
- Flawless 4K@60Hz Video with HDMI: Delivers exceptional clarity and smoothness with its 4K@60Hz HDMI port, making it ideal for high-definition presentations and entertainment. (Note: Only the HDMI port supports video projection; the USB-C port is for data transfer only.)
- Double Up on Efficiency: The two USB-A 3.0 ports and a USB-C port support a fast 5Gbps data rate, significantly boosting your transfer speeds and improving productivity.
- Fast and Reliable 85W Charging: Offers high-capacity, speedy charging for laptops up to 85W, so you spend less time tethered to an outlet and more time being productive.
- What You Get: Anker USB-C Hub (7-in-1), welcome guide, 18-month warranty, and our friendly customer service.
npm install crawlee playwrightUse a browser crawler when rendering is needed. This example reads the title after the browser has loaded the document and stores it the same way:
import { PlaywrightCrawler, Dataset } from 'crawlee';
const crawler = new PlaywrightCrawler({
async requestHandler({ request, page, log }) {
const title = await page.title();
await Dataset.pushData({
url: request.url,
title,
});
log.info(`Saved ${request.url}`);
},
});
await crawler.run(['https://crawlee.dev/js/docs/quick-start'], {
maxRequestsPerCrawl: 10,
});The browser-based version can execute page JavaScript, but it does not eliminate the need to identify the right content or decide when the page is ready. A title is available through the browser page API; for dynamic content, use an appropriate readiness condition and verify that the desired element actually exists before extracting it. Avoid adding arbitrary long delays as a substitute for understanding page behavior.
Quick wins for a faster PC:
Repair Windows errors before they cause bigger problemsFix Now →Scan for outdated or missing drivers - takes under a minuteDriver Scan →Clear out junk files and repair common Windows errorsFree Scan →Use a starter template instead
If you prefer a generated project, the official quick start offers npx crawlee create my-crawler, followed by a choice of starter template. Review the generated dependencies and request limits, then adapt its handler to your target rather than assuming a template’s selectors fit another site.
Rank #4
- Dual Converters, Infinite Potential：Includes 2× USB C male to USB A female adapters and 2× USB A male to USB C female adapters. Perfect for a wide range of uses—tablets with Bluetooth keyboards, expand USB ports on macbook, and more. Two different converters for all your daily needs
- Next-Level 10Gbps & 3A Charging: No more slow 480Mbps, this usb to usb c adapter has a transfer speed of up to 10Gbps, allowing you to do more transferring in less time. This usb adapter fits both USB A and USB C charger, supporting up to 3A fast charging
- Upgraded Exquisite Craftsmanship: With an aluminum alloy housing and metal connector, the usbc to usb adapter is extremely durable and sturdy. Rigorously tested to withstand more than 10,000 times of plugging and unplugging, ensuring long-lasting performance
- Broad Compatible: The usb c to usb adapter widely supports all USB C/ USB A devices like laptops, tablets, cellphones, car chargers, and phone chargers. Such as compatible with MacBook Pro/Air 2023/2022, Thunderbolt 4/3 Devices,Apple MagSafe Watch 9/8/7/SE/Ultra, iPad Pro 2022/2021, Samsung Galaxy S23/S20/S10, and iPhone 17/16/15 Pro. Plug and play
- Please Note: To reach 10Gbps speed, keep the cable under 3.3 ft. For USB A Male to USB C adapters, try flipping the USB C connector. USB C Male to USB A adapters support bidirectional 10Gbps transfer within 3.3 ft
How should I manage requests, sessions, and proxies?
A request limit makes a test reproducible and reduces the risk of accidentally launching an unbounded crawl. For a larger task, define the pages in scope, how links are discovered, what counts as a successful record, and how failures are logged. Crawlee’s dataset storage is useful for structured output; choose a separate persistence or deployment strategy if results must survive beyond a local run.
Crawlee also includes proxy configuration and session management. Its proxy management guide documents ProxyConfiguration integration for HTTP and browser crawler classes. Its session management guide describes SessionPool sessions that can retain cookies and proxy details; sessions can be associated with different proxy IP addresses and session-specific settings.
These features help organize crawler identity and state. They do not guarantee anonymity, successful access, or that a site’s controls will accept a request. Do not treat proxy rotation or stored cookies as a way to override access restrictions. If a target blocks or limits automated traffic, respect that restriction and seek an authorized access method.
Free tools Windows power users keep installed
One-click scans. No signup required.
Best Value
- 5-in-1 Connectivity: Equipped with a 4K HDMI port, a 5 Gbps USB-C data port, two 5 Gbps USB-A ports, and a USB C 100W PD-IN port. Note: The USB C 100W PD-IN port supports only charging and does not support data transfer devices such as headphones or speakers.
- Powerful Pass-Through Charging: Supports up to 85W pass-through charging so you can power up your laptop while you use the hub. Note: Pass-through charging requires a charger (not included). Note: To achieve full power for iPad, we recommend using a 45W wall charger.
- Transfer Files in Seconds: Move files to and from your laptop at speeds of up to 5 Gbps via the USB-C and USB-A data ports. Note: The USB C 5Gbps Data port does not support video output.
- HD Display: Connect to the HDMI port to stream or mirror content to an external monitor in resolutions of up to 4K@30Hz. Note: The USB-C ports do not support video output.
- What You Get: Anker 332 USB-C Hub (5-in-1), welcome guide, our worry-free 18-month warranty, and friendly customer service.
Where can Crawlee run?
You can run Crawlee locally or deploy it on cloud infrastructure. Apify is an optional platform path for people who want managed tooling around deployment and related workflows; Crawlee is not limited to Apify. The project README describes local and other-cloud use as well. Choose hosting based on your operational needs—such as scheduling, storage, and monitoring—rather than assuming a particular platform is required by the library.
Common Crawlee problems and fixes
- JavaScript content is missing: CheerioCrawler parses fetched HTML and does not execute client-side scripts. Confirm whether the data is in the response HTML; if not, switch to PlaywrightCrawler or PuppeteerCrawler and wait for the relevant page state.
- Playwright or Puppeteer cannot be resolved: Crawlee does not bundle these libraries. Install the matching dependency explicitly, for example
npm install crawlee playwrightornpm install crawlee puppeteer, and ensure the crawler class matches it. - Extracted fields are empty: The selector may not match the current markup, content may not have loaded, or the data may be outside the HTML being parsed. Inspect the page or response and test the selector against a known page before running a larger crawl.
- The crawl runs longer or reaches more pages than expected: Set and verify a request limit such as
maxRequestsPerCrawl; constrain link discovery to the pages in scope. - Requests fail or receive blocks: Check the site’s response, your network and crawler configuration. Proxy and session features can configure requests, but they do not ensure access. Do not increase evasion attempts; use an authorized route or stop.
- Code examples or options differ from your installation: Check the installed package version and the corresponding current API documentation. The documentation version can differ from the version pinned in your project.
Or skip the browser setup
If your task is to capture a webpage as an image or PDF rather than crawl pages and extract structured records, ScreenshotNeo offers a one-request screenshot API. For example, this cURL call saves a WebP capture of the target URL; see the API documentation for parameters and response details:
curl -G "https://api.screenshotneo.com/v1/shot" -d access_key=YOUR_API_KEY --data-urlencode url=https://crawlee.dev/js/docs/quick-start -o shot.webpScreenshotNeo removes cookie banners, newsletter popups, and chat widgets before capture; bot checks, blank pages, and failed loads are not billed. Its MCP server lets AI agents take screenshots, and the free plan includes 1,000 screenshots per month with no card; paid plans start at $5 for 3,000. This is a screenshot workflow, not a replacement for Crawlee’s multi-page extraction and crawling. Sign up free for 1,000 screenshots a month with no card.
FAQ
Is Crawlee free and open source?
The project repository identifies Crawlee as open source and licensed under Apache License 2.0. Check the repository for the license text and current project details.
What’s actually slowing this PC down?
Pick the symptom - the matching free tool is one click away.
Can I use Crawlee without Apify?
Yes. Crawlee can run locally or on other cloud infrastructure; Apify is an optional deployment path.
Does Crawlee guarantee that a scraper will keep working?
No. Site markup, loading behavior, access controls, and project dependencies can change. Crawlee supplies crawler infrastructure, but you remain responsible for selectors, validation, failure handling, and keeping your implementation aligned with the pages you are allowed to access.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
