iTechGuides is reader-supported. When you buy through links on our site, we may earn an affiliate commission. As an Amazon Associate I earn from qualifying purchases. Learn more
AI code review can be accurate and still be unhelpful. A comment may correctly identify a theoretical edge case, yet fixing it can distract from the behavior the pull request is meant to deliver. The practical answer is to define the feature’s user-facing contract first, then judge every finding by user impact and whether it should block this merge. John Lee’s 2026 essay, 209 AI Code Reviews: Accurate Is Not Necessary, illustrates the problem through one very large pull request: 209 review conversations, twelve fix-and-review rounds, and eventual convergence only after the team wrote down the intended behavior. The figures are an author-reported anecdote, not a benchmark. The durable lesson is a decision rule: correctness of an observation and necessity of acting on it are separate judgments.
What is the difference between an accurate and a necessary AI review comment?
An accurate comment describes something that is genuinely true about the code. A necessary comment identifies a change that must be made for the current product behavior, safety requirement, or merge decision. Those sets overlap, but they are not identical.
For example, an AI reviewer might correctly point out that a helper has an ambiguous name, that a defensive branch could handle an unusual state, or that a future extension would benefit from a different abstraction. None of those observations automatically belongs in the pull request under review. If the finding does not change what users receive, prevent a real failure, or violate the feature contract, making it a merge blocker can create more risk than it removes.
PC Slower Than It Used to Be?
A free scan shows the junk files, broken settings and background clutter dragging Windows down - then fixes them in one click.Free scan · Windows 10 & 11Crashes, No Sound, or Screen Glitches?
Random freezes, missing sound and display glitches usually trace back to one bad driver. Find and replace yours safely.Free scan · under a minuteJohn Lee summarizes the distinction in his essay with the sentence, “Accurate and necessary are different things.” The reviewer’s job is therefore not to maximize the number of comments fixed. It is to decide which comments matter for this change and which should be deferred or declined.
Popular on Amazon
What happened in the 209-comment case?
Lee describes a pull request that propagated secret taint across workflow steps. The change covered 83 files, added 21,962 lines, and accumulated 36 commits. Over two days, the pull request generated 209 review conversations. Both the implementation and review were produced with Codex, according to Lee’s account.
| Reported detail | Value | Qualification |
|---|---|---|
| Files changed | 83 | Author-reported pull request detail |
| Lines added | 21,962 | Author-reported pull request detail |
| Commits | 36 | Author-reported pull request detail |
| Review conversations | 209 over two days | Author-reported incident, not an audited dataset |
| Comments judged accurate | About 90% | Lee’s subjective estimate, described as “by my feel,” not a benchmark |
| Initial fix-and-review rounds | 12 in roughly two hours | Author-reported sequence |
| Final convergence | One small follow-up, then two comments in a final review request | Author-reported outcome after clarifying the feature contract |
Lee says he spent about two hours addressing twelve rounds of findings. Each round focused on the latest comment, so the code drifted toward increasingly fine-grained edge cases. The cumulative direction became harder to see: a locally sensible fix could alter assumptions elsewhere and invite another round.
The 90% figure should not be read as a measured accuracy rate for AI review. It is Lee’s personal impression of this one incident. Likewise, 209 conversations do not prove that AI review generally produces this volume. They show how a large, complex change can become difficult to steer when every true observation is treated as equally urgent.
How do I stop AI code review from becoming an endless fix-and-review cycle?
Stop treating the review feed as the project plan. Before responding to comments, write a short statement of the behavior the pull request must preserve or introduce. Use that statement as the filter for every proposed fix.
1. Write the feature contract before triage
Use one or two sentences that describe the user-visible rule and the important exception. In Lee’s case, the team wrote: credentials used only for authentication stay usable; secret values and anything derived from them are blocked automatically.
A useful contract answers three questions:
- What new behavior should users observe?
- What existing behavior must remain compatible?
- What must be blocked, rejected, or fail safely?
Keep the wording concrete enough that a reviewer can map a comment to it. “Improve security” is too broad; a rule about which values remain usable and which are blocked can guide an actual decision.
2. Read each comment through the product’s point of view
For every finding, ask what changes for a user if the suggestion is adopted and what could go wrong if it is not. This reframes review from code trivia to product impact. A comment that prevents a crash, data exposure, or incorrect workflow result deserves different treatment from a naming polish that has no runtime effect.
Quick wins for a faster PC:
Scan for outdated or missing drivers - takes under a minuteDriver Scan →Repair Windows errors before they cause bigger problemsFix Now →Fix the driver behind crashes, sound loss and screen glitchesFind Drivers →3. Check the finding against the whole change
Do not patch a comment in isolation. Identify which assumptions the proposed fix touches, whether it changes the contract, and whether it creates a new path through code that other commits rely on. A quick response can be technically correct yet trigger the next material issue because it changes the shape of the feature.
Also popular on Amazon
4. Decide, then record the decision
If the finding blocks the contract, fix it before merging. If it matters but is not required for this merge, file a follow-up issue with enough context to make the later work actionable. If it is irrelevant to the feature, explicitly ignore it. Lee describes an ignore as a judgment that the comment is not the team’s mountain, not as an admission that the reviewer was wrong.
Which AI review findings should block a merge?
Use user impact and merge relevance as the primary severity axes. Technical accuracy alone is not a severity score.
| Finding type | Question to ask | Typical decision |
|---|---|---|
| Contract violation | Does the code break a behavior the feature explicitly promises or preserves? | Fix before merge |
| Safety or failure risk | Could this cause a crash, data exposure, incorrect result, or unsafe workflow for a supported case? | Usually block and fix; confirm the supported scenario |
| Compatibility impact | Does the change break an existing caller, credential flow, or documented behavior? | Block if the compatibility requirement is in scope |
| Non-blocking correctness concern | Is the observation real but outside the current user-facing behavior? | Track as follow-up or defer |
| Maintainability or style | Would changing it reduce immediate risk, or is it preference and polish? | Defer or ignore unless the team’s conventions make it a merge requirement |
The table is a decision aid, not an automatic classifier. A crash finding in unreachable code may not block; a small-looking compatibility change may be critical. The person who understands the product has to supply that context.
Free tools Windows power users keep installed
One-click scans. No signup required.
Should I fix every accurate AI code review comment?
No. Fix every finding that is necessary to satisfy the feature contract and protect supported users; do not automatically fix every finding that happens to be true.
For each comment, write a short internal answer in this order:
- What user or system behavior does the comment affect? Name the concrete outcome, such as a blocked secret value, a usable authentication credential, or a potential crash.
- Is that behavior part of this pull request’s contract? If not, the comment may belong in separate work.
- What is the consequence of changing it now? Consider interactions with the current commits, tests, migration behavior, and compatibility assumptions.
- What is the disposition? Mark it as fix-before-merge, follow-up issue, or ignore, and state why.
This process preserves useful signal without allowing the review queue to dictate scope. It also gives the human reviewer a defensible reason for declining a technically valid suggestion.
How did a written contract change the outcome in Lee’s account?
After the twelve rounds, Lee’s team stopped responding to comments as isolated tasks and reorganized the work around the two-sentence contract. Lee reports one small follow-up and one final review request, with the work converging in two comments before merge.
That sequence is evidence from one pull request, not a controlled experiment. It does not establish that every team will obtain the same reduction or that a contract always resolves disagreement. It does show why a concise behavioral target can break a loop: reviewers gain a shared test for relevance, and implementers can reject changes that would move the code away from the intended outcome.
Also popular on Amazon
What does the human reviewer contribute that an AI model may miss?
The human supplies product purpose, supported scenarios, and the authority to choose scope. A model can inspect code and identify plausible defects, but it may not know which behavior the product intentionally permits, which edge cases are outside the release, or which compatibility promise matters most.
Lee writes, “The model can’t know that,” referring to this product context. His essay leaves open whether the product essence can be supplied to an agent up front as explicit criteria. Until that question is answered for a particular team and system, a person who understands the product must remain involved in triage.
- Product context: what the feature is for and who relies on it.
- Scope authority: which risks must be solved now and which can wait.
- Trade-off judgment: whether a proposed fix improves the release or introduces larger change risk.
- Accountability: the final decision to merge, defer, or reject a suggestion.
How should teams handle review comments that are real but out of scope?
Separate them from the merge decision without losing them. A follow-up issue should identify the affected component, the observed risk, why it is outside the current contract, and any condition that would raise its priority later.
Do these 3 things before closing this tab:
1Clear out junk files and repair common Windows errors2Fix the driver behind crashes, sound loss and screen glitches3Repair Windows errors before they cause bigger problemsDo not use a follow-up label to hide a contract violation. If the concern affects a supported user path or undermines a stated safety rule, it belongs in the current change even when fixing it is inconvenient. Conversely, leaving every style preference in the blocking queue makes serious findings harder to see.
What failure modes make AI review loops worse?
Fixing comments in arrival order
The first comment is not necessarily the most important. Arrival order rewards the review tool’s sequencing rather than user impact and can consume time before a severe finding is understood.
Measuring success by comments closed
Closing a conversation can mean the code changed, the finding was deferred, or the reviewer accepted an explanation. None of those counts alone shows that the feature is safer or more correct.
Expanding scope through defensive coding
A narrowly targeted change can accumulate abstractions and edge-case handling that were never required. Each addition increases the surface area subsequent reviewers must understand.
Recommended Free Tools
Assuming an accurate observation is automatically a blocker
This collapses truth and priority into one category. The result is a pull request optimized for theoretical completeness rather than the behavior users need now.
Also popular on Amazon
Removing product owners from review
Without someone who knows the intended behavior, the team may resolve disagreements by code volume or reviewer confidence. That can produce a technically polished change that violates the product’s actual purpose.
Independent reader supportYour contribution helps us test, update, and keep practical guides available for everyone.Can this approach be used with large pull requests?
It is especially valuable when a pull request spans many files, commits, or interacting subsystems, because local fixes are more likely to affect distant assumptions. Start by identifying the smallest contract that covers the entire change, then review comments in batches by impact rather than by file or timestamp.
For a large change, maintain a short triage list with three sections: blockers tied to the contract, follow-ups that are real but non-blocking, and declined findings with a reason. Revisit the list after substantial code movement; a previously deferred concern can become relevant if the contract or implementation changes.
What should a practical AI-review workflow look like?
- State the contract. Write the user-visible behavior, preserved behavior, and blocking or safety rule in one or two sentences.
- Inventory the review. Group comments by affected behavior instead of handling them strictly in feed order.
- Rank by impact. Put contract violations, supported-user failures, safety risks, and compatibility breaks ahead of polish.
- Inspect interactions. Before editing, trace how the proposed fix relates to current commits, tests, and adjacent paths.
- Apply the disposition. Fix blockers, create follow-up issues for non-blocking work, and ignore findings that do not serve the contract.
- Re-read the whole change. Check whether the fixes still implement the original behavior and have not introduced a new scope.
- Ask for final review against the contract. The final pass should verify the intended outcome, not reopen every theoretical possibility.
This workflow does not promise fewer comments or faster reviews in every repository. It provides a stable basis for deciding what deserves attention when comment volume becomes larger than the team’s ability to process it line by line.
What can this anecdote—and cannot it—tell us about AI review quality?
It can show a concrete failure mode: a high proportion of comments may feel accurate while the overall process still drifts away from the feature’s purpose. It can also illustrate a recovery technique based on an explicit contract and human prioritization.
It cannot establish a general accuracy percentage, a productivity improvement, or a universal number of review rounds. Lee’s 90% estimate is subjective, the incident involves one pull request, and the essay cites no independent statistical study or benchmark. Teams should measure their own outcomes if they want evidence about a particular tool, repository, or workflow.
Frequently Asked Questions
Can an AI reviewer know product intent without a human?
Only if the relevant intent is made explicit and remains complete as the system evolves. Lee’s account leaves this as an open question; in practice, supported scenarios, compatibility promises, and release scope still need a product-informed person to validate them.
Are 209 review conversations proof that AI code review is unreliable?
No. They document one author’s experience with a very large pull request, not a controlled comparison or population-level failure rate. The useful conclusion is about triage and scope, not a universal judgment on review reliability.
What should I write in a follow-up issue for a deferred finding?
Record the affected behavior, why the observation is credible, why it does not block the current contract, and what future condition would make it urgent. Include enough code or scenario context that another engineer can evaluate it without reconstructing the entire pull request.
How can reviewers disagree about necessity without arguing past each other?
Put the disputed behavior into the feature contract and name the user impact on each side. If the disagreement is about scope rather than truth, decide it as a product and release decision, then document the disposition.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
The Tool Desk
Outbyte PC Repair FREEClear out junk files and repair common Windows errorsFree Scan →Outbyte Driver Updater FREEFix the driver behind crashes, sound loss and screen glitchesFind Drivers →
