Recommended Free Tools
iTechGuides is reader-supported. When you buy through links on our site, we may earn an affiliate commission. As an Amazon Associate I earn from qualifying purchases. Learn more
Orbi’s coding-agent reviewer approved two fixes that each introduced a regression. In both cases, the agents tested the example they expected to work instead of probing the surrounding input classes. The lesson from Lawrence Liu’s small benchmark is practical: a passing review is only as strong as its test inputs, its independent oracle, and the grader used to judge it.
What the reviewer missed
Orbi’s harness is the software process around coding agents—not a physical harness. It includes role-specific prompts, loaded skills, tools available on the runner’s PATH, and rules for when review can pass. Its delivery loop takes a GitHub issue through implementation, independent review, and merge.
In the cases Liu examined, the review output for the original fork fixes said “review: pass, no findings.” Both fixes addressed a reported example but failed on nearby inputs.
What’s actually slowing this PC down?
Pick the symptom - the matching free tool is one click away.
pyinfra: text width and Unicode
A fix treated wide characters as occupying two columns, which aligned the reported example. But it produced incorrect behavior for an emoji modifier sequence and a decomposed Japanese character. The missed cases show why testing “wide characters” as one category is insufficient: Unicode sequences and combining marks can have different display behavior.
#1 Best Overall
- Careercup, Easy To Read
- Condition : Good
- Compact for travelling
chainloop: trimming secret replacements
A fix trimmed a reported secret before replacement and resolved one case. It then exposed trailing secret characters in another case and made a one-character secret match too broadly. When a patch changes text that is searched for or replaced, the review should also look for other places where that text occurs and test those collision cases.
Liu’s diagnosis was that implementers and reviewers checked examples they already expected to work instead of generating probes across the affected input classes.
How Orbi evaluated the harness
Liu ran Orbi’s delivery loop on private repository snapshots: an agent implemented an issue, opened a pull request, a separate review ran, and the change merged. A hidden grader then evaluated the result along with repository checks and tests.
Before testing harness variants, each grader had to meet three calibration checks: fail the original code, fail the faulty merged pull request, and pass a hand-verified fix. Benchmark issues were rewritten to remove links, numbers, and hints about the fix. Later runs used wrappers intended to prevent reviewers from consulting the upstream answer.
The initial tuning set contained three repositories: pyinfra, chainloop, and fedify. Variants changed prompts, skills, runner tools, implementation models, review models, or combinations. Liu cautions that many model-and-configuration cells had only two to four runs and says the deepseek-flash results are the more informative column. These are author-reported results from a small benchmark, not general performance guarantees.
What the reported results show
The benchmark separates targeted regression-prone issues from ordinary fixes. That distinction matters: a setup can handle routine localized bugs without demonstrating that it will catch regressions caused by a plausible patch.
| Benchmark set | Reported result | What it establishes |
|---|---|---|
| Three tuning issues | Deepseek-flash in both roles passed 6 of 15 runs with the original harness and 28 of 30 runs across versions v13 through v15. | Liu describes this as a training score on the issues used for tuning. |
| Nine held-out ordinary issues | Each of three tested setups passed all nine. | The issues were maintainer fixes in Go, TypeScript, and Python. The graders used maintainers’ tests and were calibrated on pre-fix and post-fix code. Liu notes these localized bugs did not require the new regression rules to pass. |
| Nine held-out ordinary issues: resource use | Version v15 used a median 5.8 million tokens per run versus 2.8 million for the original harness, and took 16 minutes versus 11. | These are the reported medians for this set. About 97% of tokens were cache reads, but the additional elapsed time was still real. |
| Three additional issues whose first upstream fix caused a regression | The original harness passed four scored runs; v15 passed three. | The v15 total excludes a completed numpy run because all three attempts were killed for memory. Its three scored runs covered aiohttp once and OpenTelemetry Collector twice. |
In an earlier comparison on nine ordinary unseen issues, v15 used a median 2.0 times the tokens and 1.45 times the wall-clock time. Those ratios describe that comparison and should not be merged with the later held-out-set medians above.
The results do not identify one universally best harness. The tuning score improved sharply, ordinary held-out performance was already perfect for all three setups, and the regression-prone set was small and did not show a clear v15 advantage. Hidden graders also have a fundamental limit: they can catch only defects someone thought to test.
What changed in the harness—and why
From examples to generated input classes
Asking agents to compare behavior before and after a change was not enough, because they chose their own examples. Moving the instruction into a skill did not fix that sampling problem. Later versions required programmatically generated inputs across relevant classes, with the number of cases recorded for review.
For text-handling changes, useful classes can include ASCII, CJK and fullwidth text, combining marks, emoji modifiers, zero-width sequences, ANSI codes, escapes, quotes, and runs of backslashes. The relevant set depends on what the patch touches; the point is to test classes systematically rather than stop at one hand-picked example.
From self-consistency to an independent oracle
Generated tests can still be circular. Liu describes a run that built a reference formatter by re-implementing the fix’s rule, then reported zero regressions. Agreement between the patch and a reference that repeats its assumption is not independent evidence that the rule is correct.
PC Slower Than It Used to Be?
A free scan shows the junk files, broken settings and background clutter dragging Windows down - then fixes them in one click.Free scan · Windows 10 & 11Outdated Drivers Are Slowing You Down
One free scan finds every outdated or missing driver and matches the right update for your exact hardware.Free scan · exact hardware matchA stronger oracle comes from an independent source, such as an existing library, known-correct prior behavior, or an invariant. List the dependencies relevant to the behavior before choosing a reference, and compare more than one reference when they disagree. Reviewers should inspect how a generated test decides what the correct answer is, not merely count the tests it ran.
Best Value
From generic conventions to repository-specific rules
Orbi also learned to use each repository’s own contribution tooling as the authority for project-specific conventions instead of hard-coding assumptions into the harness. That keeps the agent’s process aligned with the project’s actual checks.
Independent reader supportYour contribution helps us test, update, and keep practical guides available for everyone.A practical review workflow for agent-generated fixes
- Identify the behavior the patch changes. Name the input or state that triggers the issue and the nearby classes that could be affected. For a text-width change, that may include combining marks and emoji sequences as well as ordinary wide characters.
- Generate probes across those classes. Prefer programmatic test generation to a few examples selected by the implementer. Record the number and types of cases tested so the review can see the coverage being claimed.
- Check the oracle independently. Determine whether expected results come from a trusted library, known-correct prior behavior, or an invariant. If the test simply encodes the patch’s rule, it may confirm the assumption rather than validate it.
- Look for replacement collisions. If the patch changes a search term, replacement string, or matching rule, probe where else that text can occur—including short or overlapping cases.
- Use the repository’s own checks and contribution tools. Let project tooling establish conventions and run the relevant tests instead of relying on generic harness assumptions.
- Calibrate benchmark graders. Confirm that a grader rejects known-bad code and accepts a verified good fix. Keep the agent from seeing the answer when the benchmark is meant to test independent problem-solving.
For benchmark design, report tuning and held-out results separately, retain run counts alongside pass rates, and include resource costs. A strong score on issues used to refine the harness is evidence of tuning progress, not by itself evidence of generalization.
Why transcripts still matter
Automated checks can be wrong, and a benchmark can contain its own defects. Liu reports grader errors, missing snapshot files, and answer leakage. As he puts it, “A benchmark is software, and ours had bugs.” A surprising pass or failure should therefore lead to inspection of the run transcript: what the agent actually tested, which oracle it trusted, and whether the grader evaluated the intended behavior.
Quick wins for a faster PC:
Fix the driver behind crashes, sound loss and screen glitchesFind Drivers →Clear out junk files and repair common Windows errorsFree Scan →Liu’s closing rule is: “When a result surprises you, read the transcript before you believe it.”
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
