iTechGuides is reader-supported. When you buy through links on our site, we may earn an affiliate commission. As an Amazon Associate I earn from qualifying purchases. Learn more
One verification pass can uncover unsupported claims even when the numbers copied from source material are right. In a September 2026 DEV Community case study, Sumitsuke reports checking a single technical article written by AI without human editing. The author found 11 checklist issues in the initial article, fixed those findings, and also reports a contradiction introduced during the repair. The account is useful as a concrete example of what verification can catch—not as a measure of how often AI-written articles are wrong.
What the author checked—and what the counts mean
Sumitsuke describes generating one technical article from a small, fixed set of source material, freezing that output, and then applying the author’s usual verification process. The author explicitly says the sample is only one article and does not support generalization. These figures are the author’s case-specific findings, not an error rate, benchmark, or controlled comparison of AI writing systems.
The checklist covered six types of defect: incorrect facts or numbers, citations that did not support the claim, code that failed to reproduce, internal contradictions, generalizations beyond the measurements, and confusion between specification, observation, and inference.
Recommended Free Tools
| Checklist category | Found in initial article | Remaining in fixed copy |
|---|---|---|
| Wrong numbers or facts | 1 | 0 |
| Citation mismatch | 2 | 0 |
| Non-reproducing code | 0 | 0 |
| Internal contradiction | 2 | 0 |
| Generalization beyond measurement | 3 | 0 |
| Conflating specification, observation, and inference | 3 | 0 |
Sumitsuke also reports four issues involving operational rules that were not included in the writing instruction, such as disclosure and publishing-process requirements. The author treats those separately from the 11 checklist findings. The author says an initial personal pass found none of the 11 checklist findings; that observation does not establish how another reviewer would perform.
#1 Best Overall
Why correct figures did not guarantee a reliable article
All 16 figures carried over from the source material were reported correct. But one figure presented a partial breakdown as though it were a complete total. The more prominent problems, in the author’s account, came from claims that stretched beyond what the supplied material established: missing citation support, conditions omitted from claims, causal conclusions drawn from an empty search result, and conclusions broader than the range that had been checked.
This illustrates two distinct verification jobs. Checking whether a claim matches the material at hand can catch misquotation or distortion. Looking for omitted cases asks whether the available material is complete enough to support the claim in the first place. A clean match to a limited source set does not, by itself, establish a broad conclusion.
What happened during AI review and repair
The author reports three rounds of external AI review. Each round surfaced new findings as well as false positives: the reported numbers of new true findings were 3, 4, and 4, while false findings numbered 1, 1, and 2. The same false claim about a code escape sequence appeared in separate sessions and later checks. Sumitsuke says the disagreement was resolved by inspecting the exact file bytes and executing the expression. This is the author’s account, not an independently replicated result.
That episode makes reviewer confidence or repeated agreement a poor substitute for evidence in this case. The author’s concise conclusion was: “Agreement across separate sessions is not evidence.” For a code claim, the relevant evidence may be the exact source bytes and observed execution; for a factual claim, it may be the primary source and the precise conditions it reports.
Repair also needs its own check. Sumitsuke reports that one new contradiction was introduced while correcting the original findings. The number of defects that remained undetected was unknown. A list of resolved findings therefore is not proof that a document is defect-free.
How to apply the case to a verification workflow
The case supports treating automated review as a way to generate candidates for inspection, rather than as a final verdict. A practical review separates evidence checks from completeness checks and verifies changes after they are made.
- Check claims against their cited sources. Confirm that each citation supports the specific wording, including its conditions, population, timeframe, and scope.
- Separate source facts from interpretation. Mark what a specification says, what was directly observed, and what is inferred. Do not present an inference as a measurement.
- Test executable claims directly. When behavior depends on code or exact characters, inspect the actual file and run the relevant expression or example in the stated environment.
- Search for missing cases. A source match shows that a claim reflects the source; it does not show that the source covers every relevant case. Look for contrary evidence before making a broad or causal statement.
- Re-read the repaired article. Check for contradictions, changed meaning, and newly unsupported claims after edits. Keep operational requirements—such as disclosure and publishing rules—as a distinct release gate.
Sumitsuke reports 12 minutes for production and about 105 minutes for verification in this one case. Those times describe the author’s workflow for this article, not a typical time budget or a productivity comparison.
Do these 3 things before closing this tab:
1Repair Windows errors before they cause bigger problems2Scan for outdated or missing drivers - takes under a minute3Clear out junk files and repair common Windows errorsWhat this case can—and cannot—show
The case gives a detailed example of an article-level verification pass: its checklist, the kinds of defects reported, the false positives encountered during AI review, and a defect introduced during repair. It also shows why checking copied figures alone can miss problems of scope and support.
Best Value
It cannot establish a general AI error rate, rank writing or review systems, or show how often human verification will catch defects. The author’s counts come from one article, and the remaining undetected defects are unknown. The source is Sumitsuke’s first-person report, published September 17, 2026, on DEV Community.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
