Recommended Free Tools
iTechGuides is reader-supported. When you buy through links on our site, we may earn an affiliate commission. As an Amazon Associate I earn from qualifying purchases. Learn more
I wanted to know: “did any of them come?” After adding a small server-side counter, I saw requests from AI-related agents—but the most-fetched page was only a JavaScript shell, with none of the numbers an assistant might need. That changed what I built next: I put the current reading in the HTML as a real fallback, then checked what the fetched page actually returned.
These are observations from one small site, not a study or a measure of how often AI assistants fetch websites in general. The source is FirmTape’s first-person account, published August 25, 2026: FirmTape on DEV Community.
Why one “AI bot” total is not enough
A request from an AI-related user agent can reflect different activity. I separated it into three roles so the count would answer a more useful question than simply whether a bot had visited.
Free tools Windows power users keep installed
One-click scans. No signup required.
| Role | Examples in the account | What a request indicates | What it does not establish |
|---|---|---|---|
| Index | OAI-SearchBot, PerplexityBot, Claude-SearchBot | An agent is crawling on its own schedule to build a corpus. | It does not show that a person asked a question about the page or opened a result. |
| On-behalf | ChatGPT-User, Perplexity-User, Claude-User | An agent fetched a page after a person asked an assistant a question. | The fetch itself is not a human visit or citation click. |
| Training | GPTBot, ClaudeBot, Google-Extended | An agent is collecting material for model training. | It does not show indexing for search or an answer-time fetch. |
| Click-through | Attributed visits, tracked separately | A person arrived by opening a link, where attribution is available. | It is not the same event as an assistant fetching a page. |
FirmTape says ChatGPT adds utm_source=chatgpt.com to rendered links and treats visits carrying that attribution as clicks by people who chose to open a citation. That is a separate signal from a ChatGPT-User request: an on-behalf fetch can occur without a person clicking through.
How the request counter classified traffic
The account describes roughly a hundred lines of server-side counting. Its illustrative matcher identifies an agent from a user-agent string by substring. Order matters: a specific value such as chatgpt-user should be checked before a broader one such as gptbot, or a request could be assigned to the wrong category. This example is a practical matcher, not a complete or durable standard for identifying crawlers.
Server-side counting can see requests that reach the server even when no browser-side analytics code runs. Client-side analytics, by contrast, depends on a page loading and executing the analytics code in a browser; it is not a substitute for observing server requests. Neither kind of count, on its own, proves that a person read the content.
What arrived in the first short snapshot
About half an hour after shipping the counter, the author reported these totals:
| Category | Requests or visits reported |
|---|---|
| Index | 9 requests |
| On-behalf | 15 fetches, all attributed in the example to ChatGPT-User |
| Training | 0 requests |
| Click-throughs | None |
FirmTape explicitly cautions: “The honest caveat on all of this: it is roughly thirty minutes of data from a single site in a niche, and some of those fetches were me testing. It is not a study.” The counts are a limited anecdote, not a benchmark or evidence that assistants generally fetch sites at a particular rate.
The fetched page was reachable but did not contain the answer
The most-fetched URL was the site’s JavaScript application shell. Its HTML contained no numbers. A newer page the author had linked internally had not appeared in the fetched-page list. The URL receiving fetches was reachable, but the returned HTML did not contain the useful reading.
The author inferred that, in the search mode they observed, the assistant opened returned search results rather than following links from a page it had just read. That is an inference from this account, not a rule established for every assistant or mode. The practical check is narrower and more dependable: inspect the response from the URL that receives requests. A URL existing or being linked internally does not tell you what its HTML contains.
What changed in the site
FirmTape moved the current reading into a <noscript> block in the HTML shell. The author describes this as the same content served to everyone and as a genuine fallback: without JavaScript, the page otherwise would have been blank. The change addressed the actual gap in the fetched response rather than relying on the URL being discoverable.
Keep counting narrow and resilient
Aggregate by day and path
The implementation groups counts by date and path in memory, then persists them on a debounced flush. It also bounds the number of distinct paths recorded per agent per day. If writing to a paid key-value store, the author suggests persisting the accumulated day rather than writing on every request.
Best Value
Collect only what answers the question
The account’s design records the matched agent, path, and date. It does not retain IP addresses, accounts, sessions, verbatim user-agent strings, or query-string values other than the attribution parameter. Counts expire after sixty days. Those are choices made for this implementation, not a standard retention period or a privacy rule. FirmTape also says adding the tracking required an update to the site’s privacy policy.
Quick Recap
Test shutdown and missing-file behavior
- An earlier shutdown handler called
process.exit(0), which could stop a later-registered flush handler from running. - If the loader encounters a missing file, it should clear in-memory state rather than inherit old counts. The author says a test caught this load-or-reset problem.
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
