PC Slower Than It Used to Be?
A free scan shows the junk files, broken settings and background clutter dragging Windows down - then fixes them in one click.Free scan · Windows 10 & 11Crashes, No Sound, or Screen Glitches?
Random freezes, missing sound and display glitches usually trace back to one bad driver. Find and replace yours safely.Free scan · under a minuteiTechGuides is reader-supported. When you buy through links on our site, we may earn an affiliate commission. As an Amazon Associate I earn from qualifying purchases. Learn more
In one experiment, an AI reviewer named a seeded authorization bug in 9 of 10 valid diff reviews and 0 of 10 valid raw-source reviews. That is a striking difference in one small test—not proof that diffs make AI reviewers more reliable. The inputs and harness settings were not shown to be fully matched, and the tool accepted none of the 20 valid outputs as findings.
What did the experiment actually test?
Ahmad Ammar says he ran the experiment on 21 August 2026. He seeded one defect in code he wrote: mayDeleteAccount returned user.role !== 'admin', which allowed precisely the users who should not have permission to delete an account. He gave the reviewer the instruction, “Review for authorization and correctness defects,” and used a small local model referred to as local-advisor-v1:latest. The original inputs remain private, which limits independent reproduction of the reported ratios. Read Ammar’s experiment report on DEV Community.
The headline comparison is based on valid runs: the raw-source arm produced no diagnosis in 10 valid reviews, while the diff arm produced one in 9 of 10 valid reviews. There was also one crash in the diff arm, so it diagnosed the defect in 9 of 11 attempts if crashes count as failures. These are repeated runs on one seeded defect, not 21 independent bug cases.
The Tool Desk
Outbyte PC Repair FREERepair Windows errors before they cause bigger problemsFix Now →Outbyte Driver Updater FREEScan for outdated or missing drivers - takes under a minuteDriver Scan →Why doesn’t the result prove that diff formatting caused the difference?
The experiment had five arms, and the strongest results came from diff runs with a harness flag that differed from the raw-source arms. Ammar says --trust-material-paths tells the tool to treat material as a diff and extract file paths for scope checks. The report does not establish that the model received otherwise identical prompts or context in the different arms.
| Input or setting | Attempts | Valid runs | Hand-counted diagnoses |
|---|---|---|---|
| Raw source | 5 | 5 | 0 |
| Source embedded in a file | 5 | 5 | 0 |
Diff with --trust-material-paths |
5 | 4 | 4 |
| Same diff on a HEAD tree with the flag | 3 | 3 | 3 |
| Same diff without the flag | 3 | 3 | 2 |
Across the two flagged diff arms, all 7 valid runs named the defect; the unflagged diff arm named it in 2 of 3. That is consistent with a presentation effect, but it does not isolate one: the flag varied between arms, and the report does not show that all other input details were matched. Ammar estimated the prompt at 1,144 tokens with or without the flag, but did not compare the prompts themselves. Equal estimated size does not establish equal prompt content.
Did the tool accept the bug report?
No. The tool accepted 0 of 20 valid runs. Its acceptance step required a finding to name a file and line and quote evidence copied from that line. Ammar says the diagnoses appeared in free-text control and evidence fields labeled “unverified model claims,” while the accepted-findings list remained empty.
That distinction matters: the model sometimes described the defect in prose, but it did not produce a finding that passed the tool’s acceptance checks. The experiment therefore shows reported diagnoses, not 9 automatically accepted or actionable code-review findings.
What’s actually slowing this PC down?
Pick the symptom - the matching free tool is one click away.
How certain and accurate were the diagnoses?
Only one of the nine valid diff diagnoses was unhedged; the other eight expressed uncertainty. Hedging does not by itself make a diagnosis wrong, but it is relevant if a team expects confident, actionable findings rather than a suggestion to investigate.
Rank #3
An audit also corrected one false-positive label and separated an invalid-output run from completed reviews. Ammar says the corrected raw-source output described the inverted condition as a standard control that correctly limited deletion to admins—the opposite of what it did. His pattern matcher scored 7 of 11 against his hand count of 9 of 10. He cites wording differences and disagreement during hand-checking as reasons to be cautious about automated scoring. The hand-reviewed counts and scoring method should therefore be kept distinct rather than treated as a clean benchmark score.
Could input size explain the gap?
The reported materials measured 67 bytes for raw source, 390 bytes for the diff, and 1,550 bytes for source embedded in a file. The smallest and largest inputs each produced zero diagnoses, while the diff produced 9 of 10 overall. Those three sizes do not show a steadily improving relationship between input length and diagnosis, and they do not isolate length as a cause.
Ammar proposed that a diff might frame code as a deliberate change and draw attention to it. That is a hypothesis, not a measured mechanism. Because the exact original inputs are private and the prompts were not compared, the report cannot establish why the observed gap occurred.
What does the experiment say about other AI reviewers?
It does not establish that the result generalizes to other models, review tools, or codebases. The report identifies a small local model alias; a later inspection reportedly described a Qwen 3.5 architecture, 4.7 billion parameters, Q4_K_M quantization, and an 8,192-token context, but no digest was recorded during the August test to verify that those were the properties of the exact tested binary. The report also provides no evidence about frontier models.
Best Value
Nor is this an independent replication or broad benchmark. The test used one defect, and its source material is not available for exact reproduction. Ammar’s proposed follow-up with pinned frontier-model versions and matched settings was prospective in the report; it should not be mistaken for a published result.
Independent reader supportYour contribution helps us test, update, and keep practical guides available for everyone.How should you test an AI code reviewer?
Test the reviewer inside the workflow you expect to use, and separate the question “Did it notice the defect?” from “Did it produce a valid, useful finding?” A practical comparison should keep context, instructions, and harness settings fixed while varying only the input presentation you intend to evaluate.
- Prepare paired fixtures. Create a buggy fixture with a known defect and a corrected fixture. Include enough surrounding code and repository context to reflect your real reviews.
- Compare raw source and diffs fairly. Run both presentations with the same model version, instructions, context, and harness settings. If the tool has a mode or flag that changes diff handling, hold it constant where possible or test that setting as a separate variable.
- Repeat across defects. Use multiple seeded defects rather than many reruns of one issue. Repeated runs help reveal output variability, but they are not substitutes for independent cases.
- Track outcomes separately. Record diagnoses among valid outputs and successful diagnoses across all attempts, counting crashes or invalid outputs as failures for the latter. Also record whether findings are hedged, accepted by the tool, actionable, and false positives.
- Check corrected fixtures. A reviewer that finds seeded bugs but also flags correct code may add noise. Measure false positives on the corrected cases, not just recall on buggy ones.
- Review examples manually. Confirm that the reported location and quoted evidence support the claim. Keep human scoring and automated pattern matching distinct when they disagree.
If comparing tools or workflows, useful axes include detection of seeded defects, acceptance and actionability of findings, false-positive behavior, invalid-output or crash rate, and whether results persist when context and harness settings are controlled. A clean review is not a correctness guarantee, even after one seeded-defect test passes.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
