Special offer. See more information about Outbyte and uninstall instructions. Please review EULA and Privacy policy.

iTechGuides is reader-supported. When you buy through links on our site, we may earn an affiliate commission. As an Amazon Associate I earn from qualifying purchases. Learn more

Implementation is where judgments go to become invisible: a decision about what counts, when to log it, or which input to inspect can become an ordinary-looking field, test, or number. In a September 2026 essay, engineer Tom Jones illustrates how a result can be correct under its own rule yet answer a narrower question than readers think. His examples are useful as engineering cases, not as independently verified evidence of how often these failures occur.

Why an implementation decision can disappear from view

People often encounter the output of a system without seeing the choices that shaped it. A count looks like a fact; a passing test looks like assurance; a dashboard comparison looks like evidence. But each depends on implementation details: what population is included, where an event is recorded, which representation is inspected, and what a check can actually observe.

Jones’s central point is not that every implementation is biased or wrong. It is that a deliberate judgment can become encoded in routine system behavior, where later users may mistake the result for a neutral answer. Making the rule visible lets teams ask whether the result answers the question they intended.

Special offer. See more information about Outbyte and uninstall instructions. Please review EULA and Privacy policy.

What does “who is waiting?” actually count?

Jones describes building a tool to identify people waiting for his reply. It returned three counts, depending on the definition of “answered”:

Rule used Count reported by Jones What the rule includes
A reply must appear directly beneath the comment 18 Comments with a direct reply
Any later comment by Jones in the same thread counts 34 Comments in threads where he later commented, whether or not directly replying
Both rules must be satisfied 2 Comments meeting the combined condition

These are author-reported figures from Jones’s 2026 essay, not a general benchmark. None is automatically the universally correct number. Each answers a different question, so a useful report should state the definition beside the count. The same discipline applies to product analytics, alert queues, issue trackers, and any metric built from events that can be classified in more than one way.

How to tell whether a test can catch the failure it targets

A passing test provides evidence only for behavior that can occur under its conditions and that the test can observe. Jones’s examples suggest three practical checks: can the relevant path run, does the test inspect the behavior that matters, and does a deliberate break make the test fail?

Match the test environment to the failure

Jones notes that a test running against SQLite may not expose a race that can occur with Postgres, because the databases differ in write behavior. A green result in one environment cannot establish that a failure specific to another environment is absent. Use the production database engine or a suitably representative integration test when the property depends on its concurrency behavior.

Special offer. See more information about Outbyte and uninstall instructions. Please review EULA and Privacy policy.

Check reachability, observability, and failure sensitivity

If a mutation test remains green after a code change intended to break behavior, do not immediately conclude that the test suite is weak. First establish that the modified path is reachable in the test, and that the test observes the relevant outcome. Then deliberately break the behavior and verify that the test turns red. A check that never reaches the code, or cannot see its effect, cannot provide the intended protection.

Use regressions that reproduce real failures

A regression test is especially informative when it captures a failure that occurred and demonstrates that the test fails when the failure condition returns. Pascal Cescato, quoted in Jones’s essay, describes this as a test that “represents a failure that actually happened and that we proved it can fail when the condition comes back.” That does not make every regression test comprehensive, but it ties the check to an observable failure mode rather than an assumed one.

Why different checks can share the same blind spot

Having several checks does not necessarily mean having several independent lines of evidence. Jones describes gateway checks written with curl in his own input format. A vendor SDK sent a key in a different header, so the request was rejected before it reached the code path the checks were meant to assess. The checks and the system shared an assumption about request format.

Likewise, three tools that consult the same incomplete file registry can all miss the same files. Jones contrasts that kind of overlap with two tools inspecting different representations of a store: one counted 16 items and the other 2,375. Those author-reported totals revealed a scope mismatch, not proof that one tool was generally more accurate.

Special offer. See more information about Outbyte and uninstall instructions. Please review EULA and Privacy policy.

As Cescato puts it, independence is about “preserving the possibility of disagreement.” In practice, look for checks that differ in their inputs, representations, or likely failure modes—not merely in their names or codebases. A production feedback path can add a distinct perspective because it encounters real clients and formats; Cescato calls it “the one test suite whose authors we don’t control.”

What a number needs in order to keep its meaning

A metric is not just its value. Its interpretation also depends on what was counted, how cases were treated, and what comparison was used. In Jones’s retrieval examples, two setups returned 3,038 documents and 7 documents; elsewhere, 120 documents produced 86 cache keys, with 34 collisions. He also reports 2,488 requests recorded under one provider name. These figures describe specific incidents in his essay, not general rates. Their value to readers is that they prompt questions about scope, identity, and transformation: what objects entered the count, how were they grouped, and what did the labels mean?

Comparison choices can change a result’s interpretation. Jones reports 35.4% versus 30.8%, with p=0.549, under different comparison choices. The essay does not establish a universal statistical conclusion from those figures; it illustrates why a result should be accompanied by its comparison and control definition. A percentage without that context can suggest a stronger or different conclusion than the underlying analysis supports.

When publishing a metric or experiment result, state its value alongside the population, inclusion rule, treatment, comparison or control, and measurement window where relevant. These details are part of the result, not optional footnotes.

Special offer. See more information about Outbyte and uninstall instructions. Please review EULA and Privacy policy.
Independent reader supportYour contribution helps us test, update, and keep practical guides available for everyone.Support on Ko-Fi

How logging placement can distort an experiment

Random assignment is not enough if the groups are recorded at different stages. Jones describes a 68-day comparison where one group was logged at assignment, while the delivered group was logged only after later filters. The groups therefore represented different points in the pipeline, which could affect which cases entered the analysis even if assignment itself was random. Jones says the logging was fixed symmetrically.

For an experiment, trace both groups from assignment through delivery and analysis. Log comparable events at equivalent stages, and document filters that can remove or transform cases. This makes it possible to distinguish a treatment effect from a difference introduced by the measurement process.

A practical review before trusting a test or metric

  • Define the question. Write down what “answered,” “delivered,” “active,” or another key label means.
  • Make the population explicit. Identify which records, users, requests, or events qualify, and show that scope alongside counts.
  • Exercise the relevant path. Confirm the test environment can produce the targeted failure and reaches the code under review.
  • Prove the check can fail. Introduce a deliberate break or restore a known failure condition and verify the test detects it.
  • Seek genuinely different evidence. Compare inputs, data representations, and failure modes to find checks that can expose one another’s blind spots.
  • Inspect the measurement pipeline. Check grouping, transformations, logging points, exclusions, and control comparisons before interpreting a result.
  • Revisit the checklist itself. A checklist can help make assumptions visible, but it can also encode assumptions. Confirm that it asks the right question for the system being evaluated.

Source and scope

The examples and figures in this article are attributed to Tom Jones’s personal engineering essay, “Implementation is where judgements go to become invisible,” published September 27, 2026. They illustrate the essay’s argument; they are not independently audited studies or population-level statistics.

Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.

What’s actually slowing this PC down?

Pick the symptom - the matching free tool is one click away.

Special offer. See more information about Outbyte and uninstall instructions. Please review EULA and Privacy policy.