iTechGuides is reader-supported. When you buy through links on our site, we may earn an affiliate commission. As an Amazon Associate I earn from qualifying purchases. Learn more
A small Node.js crawler can turn a site into a practical first-pass audit: status codes, page metadata, headings, robots directives, images, and links, with a visible list of findings for each URL. This guide builds on a 20-check checklist and a simple scoring proposal; neither the checks nor the score are an official Google standard or a prediction of rankings.
What this crawler can—and cannot—tell you
A bounded crawler is useful when you need a repeatable snapshot of ordinary HTML pages and want to spot issues that deserve a closer look. The example approach starts at a home page, follows same-site links (including subdomains), skips non-page files such as PDFs, JPGs, and ZIPs, reads robots.txt, limits concurrency, identifies itself, and stops at a page limit. Those are practical operating choices, not universal requirements.
Its fetch-and-parse approach inspects the HTML returned by the server. It is not a full browser and does not establish what a JavaScript-rendered page looks like after execution. Google describes a crawl, render, and process sequence for JavaScript pages, and notes that rendering can be skipped when Google sees a noindex directive. A crawler that does not render pages should report what it fetched, not imply that it inspected the final browser-rendered page. Google’s JavaScript SEO basics
What’s actually slowing this PC down?
Pick the symptom - the matching free tool is one click away.
Likewise, the checklist below is a triage aid. Character counts, word counts, response duration, and document size are heuristics. They can help prioritize manual review, but do not prove that a page will rank or be excluded from search.
#1 Best Overall
- Includes Raspberry Pi 5 with 2.4Ghz 64-bit quad-core CPU (8GB RAM)
- Includes 128GB Micro SD Card pre-loaded with 64-bit Raspberry Pi OS, USB MicroSD Card Reader
- CanaKit Turbine Black Case for the Raspberry Pi 5
- CanaKit Low Noise Bearing System Fan
- Mega Heat Sink - Black Anodized
Separate crawl access from indexability
Robots.txt and noindex directives solve different problems. Robots.txt controls whether a crawler may request paths. A noindex directive is read from an accessible page’s robots meta tag or an HTTP response header. If a crawler cannot access a page, it cannot read and follow a noindex instruction on that page. Google says to use noindex or authentication to keep an accessible page out of Google, rather than treating robots.txt as an index-removal mechanism. Google’s robots.txt documentation and robots meta tag guidance
For each fetched response, record the URL requested, final URL after redirects, status, elapsed time, and relevant headers. Combine any X-Robots-Tag header with robots directives found in the HTML; a noindex response header will not appear in the document markup. Treat a blocked URL as “not fetched,” not as a page on which noindex was absent.
The 20 checks in the proposed audit
The checklist below reconstructs the 20 checks proposed by DEV Community author swiftkit_dev. The numeric thresholds and severities belong to that author’s audit method; Google’s guidance supports concepts such as descriptive metadata and crawlable links, but does not establish these exact cutoffs as ranking rules. The author’s Node crawler article
Rank #2
- Includes Raspberry Pi 4 4GB Model B with 1.5GHz 64-bit quad-core CPU (4GB RAM)
- Includes Pre-Loaded 32GB EVO+ Micro SD Card (Class 10), USB MicroSD Card Reader
- CanaKit Premium High-Gloss Raspberry Pi 4 Case with Integrated Fan Mount, CanaKit Low Noise Bearing System Fan
- CanaKit 3.5A USB-C Raspberry Pi 4 Power Supply (US Plug) with Noise Filter, Set of Heat Sinks, Display Cable - 6 foot (Supports up to 4K60p)
- CanaKit USB-C PiSwitch (On/Off Power Switch for Raspberry Pi 4)
Errors: 15 points each in the example score
- Missing title: the document has no title element.
- Noindex directive: the accessible page has noindex in a robots meta directive or X-Robots-Tag header. Report the source of the directive.
- Broken internal destination: a linked page returns a 4xx or 5xx status when checked. Record the destination and response; do not assume every failure is permanent without checking transient or access-control behavior.
Warnings: 6 points each
- Title length: fewer than 15 or more than 60 characters.
- Missing meta description.
- No H1 heading.
- Duplicate title: another crawled page has the same title. This requires comparing pages, not inspecting one page in isolation.
- Image without an alt attribute. An empty alt value can be intentional for a decorative image, so distinguish missing attributes from empty ones.
- Under 200 words: a thin-content flag under this heuristic, not a quality judgment by itself.
- Response longer than 1.5 seconds: note the measurement conditions and that a single crawl is not a stable performance benchmark.
- Missing mobile viewport tag.
- Plain HTTP: the page uses HTTP rather than HTTPS.
Notices: 2 points each
- Description length: fewer than 50 or more than 160 characters.
- More than one H1.
- Duplicate meta description: another crawled page has the same description.
- Canonical missing or pointing elsewhere: record the canonical URL and whether it differs from the fetched page. Multiple or conflicting canonical tags can produce unexpected results, according to Google. Google’s JavaScript SEO basics
- Missing lang attribute.
- Missing Open Graph tags.
- HTML larger than 1 MB.
- Redirect: record the redirect chain and final destination rather than treating every redirect as an error.
These thresholds—15/60 title characters, 50/160 description characters, 200 words, 1.5 seconds, and 1 MB—are the author’s heuristics. They are useful for sorting a report, not universal cutoffs. Google recommends descriptive titles and descriptions, crawlable links, sitemaps, and visible text content, but its cited guidance does not validate these numeric thresholds. Google’s SEO starter guide
Build the smallest useful page snapshot
For each HTML response, parse and store enough data to explain findings and compare pages later. The tutorial’s Node example uses built-in fetch and linkedom to extract these fields:
- Requested URL, final URL, status code, redirect information, and elapsed response time.
- Document title and meta description.
- H1 headings and document language.
- Canonical URL.
- Robots directives from both HTML and the X-Robots-Tag response header.
- Viewport meta tag presence.
- Images with no alt attribute.
- Links and their href destinations.
- Response body size and a word-count estimate, if you choose to apply those heuristics.
Use the response’s content type to decide whether to parse it as HTML, and keep request limits explicit. The tutorial follows redirects, uses a few concurrent requests, and applies a page cap; the right limits depend on the site and the environment. Identify the crawler honestly with a descriptive user agent and contact information where appropriate. This is a compact first pass, not a replacement for a browser-based inspection or a search engine’s own systems.
Rank #3
- Not including the Raspberry Pi 5 (8GB), the Crowpi advanced version comes with the Raspberry Pi 5
- ELECROW Black Case for the Raspberry Pi 5, CrowPi is equipped with a 9-inch HD touchscreen along with a camera; All the regular components used in DIY electronics are packed into the CrowPi development board, such as LCD, LED matrix, buzzer, light sensor, PIR sensor, ultrasonic sensor, IR sensor, etc
- Raspberry Pi Sensors: The Crowpi raspberry pi 5 programming kit is jam-packed with lots of buttons such as 19 different sensors in a tidy easy to use package; You don't have to wait and wire things
- Build Quality: Solid ABS shell and well made components in one place make it strong and convenient to travel
- Programming Lessons: This raspberry pi 5 learning kit ships with step by step instructions and provides 21 lessons to take you through identifying components reading code and running it in the terminal
Extract links in a way that reflects crawlability
Collect anchor elements with href attributes, resolve relative destinations against the final page URL, and normalize URLs before deduplicating them. Google says links are generally crawlable when expressed as an anchor element with an href, and recommends concise, descriptive, relevant anchor text. The crawler can flag anchors without destinations or links with unhelpful text for review; it should not claim that each flagged link causes a ranking penalty. Google’s crawlable links guidance
Free tools Windows power users keep installed
One-click scans. No signup required.
Keep internal-link discovery separate from optional link validation. For links not already fetched, the tutorial suggests checking once after the main crawl, trying HEAD first and falling back to GET if a server rejects HEAD with 405 or 403. Servers vary, so cap these checks and report the method and result. Avoid repeatedly requesting the same destination.
Make the score easy to recompute
The author’s proposed score starts each page at 100 and subtracts 15 points per error, 6 per warning, and 2 per notice. For example, a page with one error, two warnings, and one notice scores 100 − 15 − 12 − 2 = 71. Show the issue list and arithmetic alongside any score so a reader can verify how it was produced.
Rank #4
- Fully assembled for plug-and-play operation
- Includes Raspberry Pi 5 with 8GB RAM
- 256 GB PCIe Pi NVMe SSD (Pre-loaded with Pi 64-Bit OS)
- M.2 HAT+
- CanaKit Turbine Black Case for the Pi 5
Call this a hygiene score: it summarizes a chosen set of checks and weights. It is not a Google score, a ranking estimate, or validated evidence that one page will outperform another. The author describes it as “a hygiene score, not a guess at Google’s ranking.” Make weights configurable if the report will be used across sites with different priorities.
Independent reader supportYour contribution helps us test, update, and keep practical guides available for everyone.Report page findings and site-wide patterns separately
Some checks belong to one URL: missing title, missing viewport, a noindex directive, or an oversized response. Others only make sense across the crawl: duplicate titles and descriptions, repeated canonical destinations, redirect patterns, and recurring failures. Present the page-level result with its URL and evidence, then show cross-page groupings with the affected URL list. This prevents a reader from mistaking a duplicate detected across the crawl for a property visible on a single page.
In the 30-page example reported by swiftkit_dev, 27 titles exceeded 60 characters, 24 pages shared one meta description, 10 pages shared a title (described as versioned copies of one documentation page), one page lacked an H1, two pages were thin, and no broken links were found; the author reported an average score of 90. These are the author’s results for that particular audit, not independently verified benchmarks or typical-site statistics. The reported example and method
Best Value
- 【What you Get】You will get 1*Pi 5 8GB Single Board,1*RasTech Case,1*Active Cooler,1*Screwdriver,1*Installation instructions,12-month free warranty, lifetime service, 24-hour prompt and friendly response.
- 【More Connectors】There are two USB 3.0 ports(5Gbps simultaneously) and two USB 2.0 ports, which triple total bandwidth ,support any combination of up to two cameras or displays. Peak SD card performance is doubled through support for the SDR104 high-speed mode. It provides a smooth desktop experience for you. Offer Gigabit Ethernet and a PCIe interface, along with dual-band Wi-Fi and Bluetooth 5.0/BLE wireless capability. The RasTech Pi 5 Kit use the new 27W 5.1V 5A USB-C power connector.
- 【 Support Dual 4Kp60 Display 】Each of the two microHDMI sockets can control a 4K display at 60 Hertz, now support HDR, offering super HD video for media streaming projects. RPi 5 is the first RPi model that comes with a PCI Express port (PCIe 2.0 x1 with 500 MB/s) to attach SSDs (requires separate M.2 HAT).
- 【 Excellent Chips And Applications】Pi 5 is a full-size Pi computer using silicon built in-house at Pi. The RP1 “southbridge” provides the bulk of the I/O capabilities for Pi 5. Pi 5 is more friendly and convenient in the development of Internet of Things, Web development, machine identification, automatic control and other electronic equipment applications and network.
- 【 Faster CPU, Better GPU 】 Pi 5 features a Broadcom BCM2712 64-bit quad-core Arm Cortex-A76 processor running at 2.4GHz, it delivers a 2–3× increase in CPU performance relative to RaspberryPi 4. The 800MHz VideoCore VII GPU is compatible to OpenGL ES 3.1 and Vulkan 1.2, substantial uplift in graphics performance. Pi 5 Offers lightning-fast CPU speed, a PCI Express interface, a Real Time Clock (RTC) and a power button and runs significantly cooler than Pi 4.
Know when a small crawler is enough
A compact script is a good fit for bounded, repeatable audits where you need control over scope, request rate, and report format. Consider whether your use case needs JavaScript rendering, broader technical checks, exports, or configurable severity before choosing a more extensive tool. If rendered content matters, compare tools on whether they execute JavaScript and what they actually inspect; a simple fetch-and-parse crawler cannot answer that question for you. Google’s documentation describes why rendering can matter and why robots directives affect that process. Google’s JavaScript SEO basics
For an ongoing audit, retain timestamped crawl output and compare like with like: the same scope, request settings, and parsing rules. Keep the raw findings available so a changed score can be traced to changed pages or changed rules rather than hidden in a single number.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
Do these 3 things before closing this tab:
1Fix the driver behind crashes, sound loss and screen glitches2Clear out junk files and repair common Windows errors3Scan for outdated or missing drivers - takes under a minute
