Do these 3 things before closing this tab:
1Clear out junk files and repair common Windows errors2Fix the driver behind crashes, sound loss and screen glitches3Repair Windows errors before they cause bigger problemsiTechGuides is reader-supported. When you buy through links on our site, we may earn an affiliate commission. As an Amazon Associate I earn from qualifying purchases. Learn more
A first technical SEO audit is a controlled crawl of a site, followed by a check of what the crawl found and a short list of problems you can prove. You can run it with Screaming Frog SEO Spider, KWT Spider or Sitebulb. The steps are the same in all three: agree the scope, run a small sample crawl, decide whether JavaScript rendering matters, add URL sources the links miss, review the reports in a fixed order, verify each issue, and then turn the verified problems into prioritized actions.
A crawler only reports what its configuration can discover and fetch. Scope, robots rules, URL and depth limits, and rendering all change what appears in the results, so the first audit is as much about setting the crawl up correctly as it is about reading the output.
Step 1: Agree scope and permission before you crawl
Confirm the site, the start URL and the owner’s approval before opening any crawler. A crawl sends many requests to a live server, so an unplanned crawl can slow a site or trigger alerts. Write down the following before you start:
- The exact start URL, and whether the protocol and the www or non-www version are the canonical host.
- The in-scope subdomains, and any staging, test or development areas to exclude.
- Any known URL lists the owner wants included, such as a list of landing pages or a product feed export.
- Crawl constraints: request speed, maximum URLs, and whether the owner allows crawling during business hours.
- The deliverable: a prioritized issue list, a fix ticket queue, or a report for a client or manager.
Keep robots rules enabled unless the owner has explicitly approved otherwise. Screaming Frog’s default configuration respects robots.txt, and KWT Labs recommends leaving “Respect robots.txt” enabled. Both are documented in their settings guides: Screaming Frog SEO Spider configuration and KWT Spider crawl settings.
Choosing among Screaming Frog, KWT Spider and Sitebulb
The three tools differ in platform, JavaScript handling, crawl controls and report style. The table below uses the vendors’ own documentation, so treat it as a list of documented features rather than a performance comparison. Confirm current prices and limits on each vendor’s site before you buy.
| Comparison axis | Screaming Frog SEO Spider | KWT Spider | Sitebulb |
|---|---|---|---|
| Platform and product form | SEO Spider desktop software. The free Lite version crawls up to 500 URLs, according to Screaming Frog’s canonical-audit tutorial (accessed 2026). | Windows desktop technical SEO crawler, as described by KWT Labs in its v1.3 documentation (updated June 26, 2026). | Desktop and Cloud offerings. The vendor homepage states up to 500,000 URLs per Desktop audit and up to 10 million URLs per Cloud audit (accessed 2026). These are plan capacities, not independent performance tests. |
| JavaScript handling | The standard configuration does not execute JavaScript. Its FAQ says JavaScript rendering is available in the paid version. | The v1.3 FAQ describes basic rendering. Complex single-page applications may expose fewer URLs than a full browser crawl. | An HTML Crawler for quick traditional extraction and a Chrome Crawler that renders pages with headless Chrome. Support recommends the Chrome Crawler when JavaScript-rendered content matters. |
| Crawl setup controls | Crawl behavior, robots.txt handling, canonical checks and sitemap crawling are documented. | v1.3 documentation lists maximum depth, maximum URLs, threads, robots.txt handling and user agent. | Setup lets you select audit data, crawl sources, crawler type, subdomain options and speed limits. |
| Reporting style | Tabs and filters show raw crawl data and issue-specific views. | Overview and detail tabs, with export formats described by KWT Labs. | Prioritized Hints, contextual reports and visualizations are the emphasis. |
| Good fit when | You want configurable raw crawl data and the free 500-URL entry point covers your scope. | You need a Windows desktop tool and the v1.3 features cover your requirements. | You prefer an audit-led report workflow, or you need to compare HTML and rendered data. |
Useful sources for each product: Screaming Frog SEO Spider FAQ, KWT Spider FAQ, and Sitebulb’s product page.
Step 2: Run a small sample crawl first
A small sample is the safest way to learn the interface and to catch a wrong start URL or a bad configuration before a full crawl. KWT’s getting-started guidance suggests starting with a small site or limiting depth to 3 while you learn. Sitebulb’s setup guide also describes a sample crawl for very small sites. Use the same approach in any tool:
Recommended Free Tools
- Set the start URL to the agreed host and confirm that the first response is the expected page, not a redirect loop or a login wall.
- Set a low URL limit, for example a few hundred URLs, and a shallow depth limit, such as 3.
- Keep robots.txt handling on, and set a moderate request speed or thread count so the server is not overloaded.
- Run the crawl and check that the crawl finished, that the URL count is plausible for the site, and that the response codes look sensible.
- Only then raise the URL, depth or speed limits for the full crawl, and record each setting so the audit can be repeated.
Tune limits to the site and the permission you were given. Vendor defaults are starting points, not best practice. KWT’s v1.3 documentation gives a default maximum of 100,000 URLs, which is a software default and not a recommendation for every site.
Rank #2
Step 3: Decide whether JavaScript rendering matters
Many sites serve complete HTML, and for those a plain HTML crawl is faster and sufficient. Rendering matters when important content or navigation appears only after JavaScript runs, such as product listings loaded by a script, menus built client-side, or links that exist only after an interaction. Sitebulb’s crawler settings documentation recommends the HTML Crawler for most sites and the Chrome Crawler for JavaScript frameworks or rendered content. Its support guide notes that Chrome downloads page resources and takes longer. Read the Sitebulb crawler settings guide for the current options.
A sample comparison test
Pick five to ten representative pages, such as the homepage, a category page, a product page, an article and a paginated list. Run one HTML crawl and one rendered crawl over the same URLs, then compare:
- The internal links found on each page in both crawls.
- The page titles, meta robots directives and canonical tags that each crawl reads.
- Whether the rendered crawl finds URLs that the HTML crawl misses.
If the two crawls agree, the faster HTML crawl is enough for the audit. If the rendered crawl finds important URLs or content that the HTML crawl does not, use a rendered crawl for the full audit and note the mode in your report. Be aware that a rendered crawl can still miss some links in complex single-page applications, so check the results against the live site and the sitemap.
Step 4: Add URL sources beyond internal links
A link-following crawl starts from the start URL and follows links it can see. It will miss orphan pages, which are pages with no internal links pointing to them. To find more of the URL set, add discovery sources where you are authorized to use them:
- XML sitemap URLs.
- Analytics landing-page exports, where the owner agrees.
- Search Console URL data, where the owner agrees.
- An approved list of URLs from the owner.
Sitebulb’s setup guide supports XML sitemap, Google Analytics and Google Search Console as crawl sources. Screaming Frog’s sitemap tutorial explains how to match crawled URLs against sitemap URLs, which reveals two groups: pages that appear only in the sitemap, and pages that the crawl found but the sitemap does not list. See Sitebulb’s Technical SEO Auditing crawl setup guide and Screaming Frog’s sitemap audit tutorial.
Sitebulb’s guide puts it this way: “Carrying out a successful and efficient Technical SEO Audit starts with gathering the correct audit data.” The guide is a vendor publication dated July 27, 2026, and it does not name an individual author.
Step 5: Review the reports in a fixed order
Work through the crawl in the same sequence every time. Earlier checks tell you whether later checks can be trusted. Start with coverage, then move to indexability and crawlability, canonicals, sitemap comparison, and finally on-page and linking details.
Crawl coverage and response status
- Confirm the crawl finished and the URL count is close to what you expected from the sample.
- Group URLs by response code. Note the share of 3xx redirects, 4xx errors and 5xx server errors, and sample each group.
- Check redirect chains and loops, and whether internal links point to URLs that redirect.
Indexability and crawlability
Sitebulb’s indexability guide places indexability and crawlability early in the audit and points to reports for indexable, not-indexable, nofollow and disallowed URLs. Use the equivalent filters in Screaming Frog or KWT Spider. Look at robots.txt blocks, noindex directives, nofollow directives and status codes together, because a URL can be blocked for one reason and noindexed for another. The Sitebulb indexability and crawlability guide explains how each category is defined.
Canonicals
A canonical tag is a signal to search engines about the preferred version of a page. It is not a command, and its presence does not prove it is correct. For each canonical group, check that the canonical target returns a 200 response, is indexable, and points to a page that is itself the preferred version. Screaming Frog’s canonical audit tutorial describes this workflow.
Sitemap comparison
Compare the sitemap URLs with the crawled URLs. Sitemap URLs that return errors, redirect, or are noindexed are candidates for cleanup. Crawled indexable URLs that are missing from the sitemap are candidates for review. Do not assume the sitemap is complete or correct without checking a sample.
Internal linking and page elements
Review titles, meta descriptions, headings and internal link counts, and look for pages with few or no inbound internal links. Treat these as leads to check on the live pages, not as proof of a problem.
Free tools Windows power users keep installed
One-click scans. No signup required.
Step 6: Validate before you call anything a defect
Crawler warnings are leads for investigation. Before you report an issue, confirm it:
Best Value
- Features Over 160 Latin Songs
- Arranged for C Instruments
- Standard Notation
- 48 Pages
- Open three to five affected URLs in a browser and compare what you see with what the crawler reported.
- Check whether the issue belongs to a template, such as every product page, or to individual pages.
- Recrawl the sample URLs with rendering on if the page depends on JavaScript, and compare the results.
- Record the evidence: the URL, the report name, the value found and the time of the crawl.
- Keep a separate list of tool flags you could not confirm, so they do not appear as defects.
Validate a template before you recommend a sitewide change. A single broken canonical on one page is a fix request. The same pattern on a template that generates thousands of pages is a larger change that needs owner approval and a test plan.
Step 7: Turn verified problems into prioritized actions
Each finding in the final list should include the following fields, so the owner can act on it without rerunning the audit:
- Affected URL pattern or template, with example URLs.
- Evidence: the report name, the value found, and the validation step you took.
- The consequence for users or search engines, only where the evidence supports it.
- The recommended fix, written as a specific change.
- The owner who can make the change.
- A priority, based on how many important URLs are affected and how much the issue blocks crawling, indexing or user journeys.
Order the list by priority and by effort, and keep the wording plain. “Canonical targets on product variant pages return 404 on 214 sampled URLs in a rendered crawl” is actionable. “Lots of canonical errors” is not. Numbers in the list should always carry their scope, such as the crawl date, the mode, and whether the sample covers the whole template.
Quick wins for a faster PC:
Scan for outdated or missing drivers - takes under a minuteDriver Scan →Repair Windows errors before they cause bigger problemsFix Now →Limits to keep in mind
Some reports can be omitted on very large sites to save time. Sitebulb’s setup guide notes that omitting some reports on sites with 100,000 pages or more reduces crawl time, but it also means missing checks such as image alt text, page speed and mobile friendliness. If you omit them, say so in the report.
A first audit cannot tell you how much an issue affects rankings. It can show what a crawler found, what a browser shows, and what the owner can verify. Describe fixes as changes to crawlability, indexability, linking or page quality, and let the owner judge the expected effect with their own data.
The Bottom Line
For a first audit, start with the tool your site and permission suit, not the one with the most features. Screaming Frog suits teams that want raw, configurable crawl data. KWT Spider suits Windows users who need its v1.3 controls. Sitebulb suits teams that want an audit-led report and a direct HTML versus rendered comparison. Whichever you choose, keep the scope written down, run the sample first, and count only issues you have verified.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
