PC Slower Than It Used to Be?
A free scan shows the junk files, broken settings and background clutter dragging Windows down - then fixes them in one click.Free scan · Windows 10 & 11Outdated Drivers Are Slowing You Down
One free scan finds every outdated or missing driver and matches the right update for your exact hardware.Free scan · exact hardware matchOne developer says a crawl project that began on a 2 GB VPS grew to 38.5 million known domains. The figure describes domains discovered, not 38.5 million fully indexed websites: the project reported 24.8 million indexed pages, with its early strategy emphasizing homepages. In a September 2026 retrospective, the author, writing as cl0q, explains how the crawl expanded—and why storage and database contention proved harder than the headline scale suggests.
What does “38.5 million domains” actually mean?
In a retrospective published September 27, 2026, cl0q reported 38.5 million known domains in the project database. The same account listed 24.8 million indexed pages, of which 24.4 million were live indexed pages. These are self-reported project figures, not independently audited measurements. The author said 97% of known domains had been attempted and 92% resolved in DNS; neither figure means every page on those sites was crawled or searchable.
The crawl began in 2025, with its first recorded crawl in April. Its initial goal was broad discovery, and the author said the crawler fetched homepages while aiming to expand into deeper coverage later. That distinction matters: domain discovery, successful DNS resolution, a crawl attempt, and an indexed page are different milestones.
| Project measure | Figure reported by cl0q in 2026 | What it indicates |
|---|---|---|
| Known domains | 38.5 million | Domains in the project database, not a count of fully indexed sites. |
| Indexed pages | 24.8 million | Pages indexed; the account does not describe this as complete site coverage. |
| Live indexed pages | 24.4 million | Pages the author classified as live in the project database. |
| Known domains attempted | 97% | Reported share with a crawl attempt, not a guarantee of a successful or comprehensive crawl. |
| Known domains resolving in DNS | 92% | Reported share resolving in DNS. |
| TLDs represented | 1,027 | Reported breadth of top-level domains. |
| Pages crawled per day | About 250,000–300,000 | Reported daily crawl volume, not an independently measured rate. |
All figures in the table come from cl0q’s September 27, 2026 account and are project-reported rather than independently verified. The author said the counts were measured from the project database, rather than taken from its dashboard. Read the author’s retrospective on DEV Community.
Quick wins for a faster PC:
Scan for outdated or missing drivers - takes under a minuteDriver Scan →Repair Windows errors before they cause bigger problemsFix Now →Fix the driver behind crashes, sound loss and screen glitchesFind Drivers →#1 Best Overall
- WHY CHOOSE G3 ULTRA MINI PC PENTIUM GOLD 7505 - Choose the Intel Pentium Gold 7505 for snappier everyday responsiveness: It delivers up to 30% faster single-core performance than the Ryzen 5 3500U, making office apps and web browsing feel noticeably quicker, while its Intel UHD Graphics (48 EUs) provides 2.4x the GPU performance of the N100 & N150's 24-EU graphics, ensuring smoother 4K streaming and light photo editing.
- 16GB RAM MEMORY & 512GB STORAGE - GMKtec Nucbox G3 Ultra mini computer is prebuilt with 16GB LPDDR4 RAM at 3200 MT/s, you will enjoy a speedier experience with Built-in 512GB M.2 SATA Hard Drive. Our mini desktop pc boots up in seconds, work on multiple browser tabs, software applications and quickly transfers files. There is a primary slot and secondary expansion storage. Primary slot is M.2 2280 PCIE and secondary slot is M.2 2280 SATA.
- RICH INTERFACE - Nucbox pentium mini computer is equipped with 3* USB 3.2 Gen2 ports, up to 10Gbps/S, 1*USB 2.0, HDMI(4K@60Hz)*2, 3.5mm Audio Jack. Supports WiFi 6, and Gigabit Ethernet RJ45 2.5GbE network connectivity, Bluetooth 5.2. This Mini PC supports multiple device connection and can be used with servers, monitoring equipment, office equipment, displays, projectors, televisions, etc.
- 4K DUAL SCREEN DISPLAY - Mini desktop computer is equipped with upgraded Intel Graphics(max 1000MHz), supports 4K video playback and AV1 decoding, connect the pc with a projector as a home theatre, enjoy a variety of entertainments. Two HDMI 2.0 ports allows you to multi-task efficiently on two 4K@60Hz displays.
- UPGRADED COOLING FAN - The G3 Ultra has upgraded the cooling fan to reduce fan noise and thermals. We are using an upgraded thermal paste as well to help reduce heat on the CPU.
How did the crawl discover so many domains?
A naive link strategy created a skewed crawl
At first, following links led the crawler into roughly a million Blogspot subdomains associated with one domain, according to cl0q. That is a practical failure mode for web discovery: a large number of apparent hosts can dominate the crawl without delivering equivalent breadth across the web.
A rotating frontier improved breadth
The author says they shifted to a rotating frontier: repeatedly sample domains already known to the project and mine their homepages for outbound links. Certificate Transparency (CT) logs offered another discovery source. At peak, cl0q reported surfacing around 1,700 new domains a minute from CT logs. This is a peak discovery figure in the author’s account, not a sustained crawl rate or a measure of how many discovered sites became indexed.
The trade-off is between breadth and depth. A crawler can find many candidate domains without visiting many pages on each one. The project’s reported homepage-first approach favored getting a broad map of hosts; deeper coverage would require further crawling and more storage and database work.
What did the system look like?
Cl0q describes four small, incremental stages, designed to be restartable rather than dependent on one enormous run:
Rank #2
- 12th Intel Alder Lake N95 Processor – The GMKtec G3 S Mini PC is powered by the 12th Gen Intel N95 processor with 4 cores, 4 threads, 6MB cache and a burst frequency up to 3.4GHz. Compared with N100/N5105/N5100/N5095, the N95 delivers up to 36% overall performance improvement. Perfect for routine tasks, office work, and home entertainment, this compact mini desktop is more convenient than traditional bulky PCs.
- 8GB RAM & 256GB SSD Storage – Pre-installed with 8GB DDR4 memory and a fast 256GB M.2 2242 SSD, the G3 S mini desktop offers quicker startup, smoother multitasking, and faster file transfers. Enjoy seamless performance whether you’re working on multiple applications, browsing, or streaming content.
- Rich Interfaces & Connectivity – The G3 S mini computer comes equipped with USB 3.2 (up to 10Gbps), dual HDMI 2.0 (4K@60Hz), and a 3.5mm audio jack. With support for WiFi 5, Bluetooth 5.0, and Gigabit Ethernet (RJ45 1000MbE), it connects easily with monitors, projectors, printers, office equipment, and other peripherals, making it versatile for both home and business use.
- Dual 4K Display Support – Featuring upgraded Intel UHD Graphics (up to 1000MHz), the G3 S supports 4K video playback and AV1 decoding for a smooth viewing experience. With dual HDMI outputs, you can connect two 4K@60Hz displays simultaneously, enabling efficient multitasking for work and entertainment.
- GMKtec WARRANTY - GMKtec offers a 1-year limited GMKtec's warranty for each mini PC, starting from the date of the purchase. All defects due to design and workmanship are covered. With a professional after sales team always ready to attend to your needs, you can simply relax and enjoy your mini PC.
- Discovery: find candidate domains through the frontier and sources such as CT logs.
- Page crawl: fetch pages from discovered domains, with homepages central to the reported early approach.
- Enrichment: process crawled data, including favicon handling.
- Serving: make indexed results available through the search service.
Breaking work into stages meant interrupted jobs could resume incrementally, according to the author’s description. The account does not provide enough detail to treat this as a general performance benchmark or a complete implementation guide.
Why was a 2 GB VPS not the whole infrastructure story?
Disk filled up faster than expected
The initial 2 GB VPS is the memorable headline, but the author identifies disk capacity as a major constraint as the project grew. Moving storage to a remote host connected over Tailscale did not make storage cease to be a bottleneck. The lesson is not that 2 GB is sufficient for a production-scale search index; it is that the project began on a very small machine and expanded its setup as needs changed.
Cl0q put the storage warning bluntly: “If there’s one thing I’d tell anyone starting a crawl project: budget ten times the disk you think you need, then double it.” — cl0q, project author.
Search and crawling competed for database time
The project’s search and crawl workloads shared a single-core database and table, the author says. Background jobs and the crawler’s target selection could slow search. This exposes a basic tension in a small search system: the same database capacity must support both interactive queries and ongoing ingestion. A system can discover or fetch more material while becoming less responsive to people searching it.
The Tool Desk
Outbyte PC Repair FREERepair Windows errors before they cause bigger problemsFix Now →Outbyte Driver Updater FREEScan for outdated or missing drivers - takes under a minuteDriver Scan →Rank #3
- ➊ [ Trusted Quality for Everyday Agentic AI ] GEEKOM equips its SSDs with reliable original-grade flash and conducts rigorous stability testing to support dependable everyday operation. This commitment to quality is backed by a 3-year warranty. Simply connect the Air12 to cloud AI services for research, writing, study support and daily productivity—no NPU or complex local setup required. Designed for students, home users, light office work and first-time buyers, the Air12 is a high-value Cloud Agentic PC for everyday tasks
- ➋ [ Intel 7505 processor ] Powered by the Intel 7505 processor (2 cores, 4 threads, up to 3.5GHz), the GEEKOM Mini PC Air12 delivers smooth performance for everyday computing, office tasks, and home entertainment. With enhanced single-core processing, it handles daily workloads efficiently and responsively. Compact, quiet, and energy-efficient — a solid alternative to bulky desktops.
- ➌ [440lbs(200kg) Pressure Rated Metal Frame for Demanding Environments] Unlike the Plastic Shells You’ll Find on Most Mini PCs, geekom Mini Air12 features a triple-reinforced ABS+PC shell, precision-crafted metal frame and baseplate—engineered to withstand up to 440 lbs of pressure for the perfect balance of strength and thermal efficiency. Tool-free upgrades, shock-absorbing feet, and a 3D antenna deliver true durability
- ➍ [Dual-Channel RAM & NVMe SSD Expandability] Ships with 8GB DDR4 RAM and a 256GB NVMe SSD for smooth everyday performance. Dual memory slots and dual storage slots give you the flexibility to upgrade to 64GB RAM and 2TB SSD, so your system can adapt as your workload grows. Enjoy faster load times, smoother multitasking, and long-term reliability.
- ➎ [Triple 4K Displays for Maximum Productivity] Connect up to three 4K monitors via HDMI 2.0, Mini DisplayPort 1.4, and USB-C — ideal for stock trading dashboards, multi-tab research, office document editing, and light spreadsheet work. WiFi 6 and Bluetooth with high-gain antenna ensure stable wireless connections throughout your workspace. 5x USB ports and a full-size SD card reader provide quick access to peripherals and camera files — no adapters required.
The retrospective frames the project’s main engineering pressure as disk usage and database contention—not simply how quickly a machine could fetch pages. It does not establish that the setup would perform similarly under other workloads or on different hardware.
Why reject some pages instead of indexing everything?
The crawler filters anti-bot interstitials rather than treating their challenge text as ordinary page content. That choice protects the usefulness of results: indexing a block page can make a search result appear to represent a site’s content when it actually represents a barrier to automated access.
Cl0q summarized the principle this way: “A search engine is only as good as what it refuses to index.” — cl0q, project author. It is a quality-control rule, not a claim that the crawler can identify every misleading or low-value page.
Independent reader supportYour contribution helps us test, update, and keep practical guides available for everyone.What do favicon counts tell us—and what don’t they tell us?
Cl0q reported that 16.3 million sites had favicons, which the project reduced to about 7 million distinct icons using SHA-256 hashes. Grouping identical icons can offer an investigative pivot: sites sharing an icon may merit further examination. But a shared favicon is only an association, not proof that sites have the same owner, operator, or infrastructure.
Rank #4
- 【Ryzen 5 3500U Processor】KAMRUI Essenx E2 Mini PC is equipped with AMD Ryzen 5 3500U (4-cores/8-threads, up to 3.7GHz) with integrated Radeon Vega 8 Graphics(1200MHz, 8 Core). The 3500U CPU operates at a base frequency of 2.1 GHz and a Boost frequency of 3.7 GHz. This DDR supports upgradable up to 32GB, SSD supports up to 2TB.(NOT INCLUED), KAMRUI E2 3500U Mini PC is ideal for light office work and home entertainment. KAMRUI E2 3500U is more than 35% more powerful and smoother in operation than the Intel N150, 33% faster than Intel N95, 28% performance boost over Intel i3-10110U, and 42% stronger processing power than AMD Ryzen 3 3200U.
- 【16GB DDR4 & 256GB SSD】The KAMRUI E2 mini computers is equipped with 16GB DDR4(Expandable up to 32GB) for faster multitasking and smooth application switching. 256GB M.2 SSD ensures fast startup times,fast file transfers and plenty of storage space,eliminating slow loading times and ensuring fast responsiveness.Storage space can RAM supports up to 32 GB, SSD supports up to 2TB (Not included)make file storage easier.
- 【4K Dual Display & USB 3.2 Type-A Port】KAMRUI E2 3500U mini desktop pc is equipped with an HDMI 2.0+DP 1.4 interfaces for faster transmission, Support Dual 4K@60Hz Display, E2 mini desktop computers is ideal for visual home entertainment, home office, conference rooms, etc. USB3.2 Gen1 Type-A Port×2 with a transfer speed of up to 5Gbps (10 times faster than USB 2.0) for efficient data transfer. The RJ45 1000M Gigabit Ethernet Port ensures a stable network connection.
- 【WiFi+Bluetooth stable connection】The Kamrui E2 micro pc have reliable and stable wireless connection, open websites in seconds, watch movies without buffering and download files smoothly, connect your monitor from WiFi or Ethernet, use a wireless keyboard and mouse through bluetooth, which will be powerful workstation for you.
- 【Versatile Ports】This KAMRUI E2 Small pc is equipped with HDMI 2.0×1(4K@60Hz)、DP1.4×1(4K@60Hz)、Gigabit Ethernet Port (RJ45, 10/100/1000Mbps) ×1、USB3.2 Gen1 Type-A Port×2（5Gbps）、USB2.0 Type-A Port×2、3.5mm Audio Jack ×1、DC In ×1、Power Button ×1
What did the project promise about privacy and openness?
In the September 2026 account, cl0q said the search engine set no cookies and stored no searches for logged-out visitors. Accounts were described as optional and passkey-based; search history existed only when enabled. These are the project author’s descriptions, not an independent privacy audit or confirmation of behavior today.
The author characterized the project as open source but said its public source mirror was a couple of months behind at the time of writing. That caveat makes it difficult to treat the mirrored code as a real-time record of the deployed service. The account also mentioned a free beta API and anticipated paid tiers, but those were plans described at publication, not evidence of present availability or terms.
What should readers take away from the headline?
- It is a domain-discovery milestone, not a coverage guarantee. The 38.5 million figure counts known domains; the author separately reported page counts and said the crawl focused on homepages.
- Discovery needs safeguards against skew. Link structure can flood a frontier with many related subdomains, so the project changed how it rotated through known domains and added CT logs as a discovery source.
- Storage and shared database capacity shape the service. A small, restartable pipeline can still run into disk limits and contention between indexing work and search queries.
- Indexing is a selection problem as well as a scale problem. Excluding anti-bot challenge pages is one example of refusing content that would make results less useful.
Cl0q’s stated target was 100 million domains. The author also said recrawls would start October 9, 2026, but the September 27 post only records that as a future plan; it does not establish whether the recrawl began. The same applies to later API, pricing, source-mirror, and count changes: the retrospective is a snapshot of what the author reported then, not a current status report.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
