Do these 3 things before closing this tab:
1Scan for outdated or missing drivers - takes under a minute2Repair Windows errors before they cause bigger problems3Fix the driver behind crashes, sound loss and screen glitchesiTechGuides is reader-supported. When you buy through links on our site, we may earn an affiliate commission. As an Amazon Associate I earn from qualifying purchases. Learn more
An LLM critic can flag and explain possible code-review issues, but it should not be the final authority on whether a build fails. Use constrained labels, then let deterministic code check those labels against an explicit set of fatal findings. In Python, a frozenset is a useful way to keep that policy fixed at runtime. It stabilizes the rule—not the critic’s judgment.
Why can an LLM judge give different answers?
LLM judges can be inconsistent, including when asked to compare two responses. A 2025 empirical study in software engineering found that pairwise judgments often changed when the response order was reversed; the inference strategies tested did not fully eliminate the inconsistency. The finding is specific to the study’s tasks and methods, but it is a practical warning against treating one model response as a dependable pass/fail decision. Wang et al., “Can LLMs Replace Human Evaluators? An Empirical Study of LLM-as-a-Judge in Software Engineering” (June 22, 2025).
Some evaluation tasks also have more than one reasonable answer. Microsoft Research’s NeurIPS 2025 study calls this rating indeterminacy. Across 11 rating tasks and eight commercial LLMs, the authors reported that forced-choice validation approaches selected judge systems performing as much as 30% worse than their response-set approach. That result is limited to the study’s tasks and methods; it does not mean every forced-choice rubric will perform worse. Microsoft Research, “Rating Indeterminacy in LLM-as-a-Judge” (2025).
Adding judges is not automatically a solution. In Apple Machine Learning Research’s tested natural-language-inference datasets, a panel of nine frontier LLMs from seven model families provided about as much information as two independent votes. Correlated errors limited the benefit of a larger panel. The result is specific to those datasets and models, but it illustrates why aggregation cannot compensate for judges that share the same blind spots. Apple Machine Learning Research (2026).
#1 Best Overall
Separate the critic’s judgment from the failure policy
Give the model a defined job: identify findings from a controlled vocabulary and provide evidence or a rationale. Give your program a separate job: decide which labels are fatal. That boundary makes the acceptance rule inspectable and testable even when the model’s classification is not perfectly repeatable.
- Define the labels. Choose a small vocabulary that describes actionable findings, such as
security_issue,data_loss, andstyle_issue. The examples are illustrative; your labels should match your project’s review policy. - Constrain the critic’s output. Request structured output with labels from that vocabulary, plus a rationale and relevant code evidence. Treat an unknown label or malformed response as something to review—not as an automatic pass.
- Mark fatal labels in code. Keep the fatal subset separate from the full vocabulary so policy changes are deliberate and visible.
- Fail only on a recognized fatal label. Parse the output, intersect its labels with the fatal subset, and fail the build only if that intersection is non-empty.
- Retain and audit the result. Preserve the raw response, parsed labels, rationale, and decision. Route uncertain, malformed, or novel cases to a human reviewer.
Use a frozenset for the fixed fatal vocabulary
Python’s official 3.14.8 documentation describes frozenset as immutable and hashable. Its contents cannot be changed after creation, so it can also serve as a dictionary key or as a member of another set. Ordinary sets are unordered collections of distinct hashable objects. For fatal-label checks, membership does not depend on item order, and immutability helps prevent accidental edits to the configured policy while the program runs. Python 3.14.8: Set Types — set, frozenset.
ALLOWED_LABELS = frozenset({
"security_issue",
"data_loss",
"style_issue",
})
FATAL_LABELS = frozenset({
"security_issue",
"data_loss",
})
def decide(parsed_labels):
labels = set(parsed_labels)
unknown = labels - ALLOWED_LABELS
if unknown:
return "review", {"unknown_labels": unknown}
fatal = labels & FATAL_LABELS
if fatal:
return "fail", {"fatal_labels": fatal}
return "pass", {"fatal_labels": set()}
This sketch assumes parsed_labels has already been extracted from a response. In production, validate the response format and types before calling the decision function; do not silently coerce arbitrary model output into trusted labels. A real system should also record why a label was assigned and define what happens when the critic is uncertain or unavailable.
What this design does—and does not—guarantee
- It makes the configured fatal categories stable during execution. The frozen collection cannot be mutated in place, so unrelated code cannot accidentally add or remove a fatal label through a set operation.
- It makes the final gate deterministic for the same parsed labels and policy. If the parsed labels differ between runs, the result can still differ.
- It does not verify the critic’s classification. A missed security issue can still pass if the model fails to emit the relevant label. A false positive can still fail if the model emits one.
- It does not prove the policy is complete. The fatal vocabulary needs review against your project’s risks and real examples.
- It does not remove shared bias among multiple judges. A panel can agree and still be wrong, particularly when its members make correlated errors.
For that reason, evaluate classification quality separately from gate logic. Build a set of representative examples with human-reviewed expected labels, including borderline cases, and measure false positives and false negatives. Revisit the examples and policy when the codebase, threat model, rubric, or model changes.
Make the rubric and escalation path part of the system
A rubric should explain what each label means, what evidence qualifies, and how to handle uncertainty. A 2026 Google Research automated-patch-repair study illustrates a review process in which an LLM drafted a rubric, an expert refined a “golden” rubric, and an LLM judge assessed patches against it. The evaluation covered 48 bugs and 115 patches; in that setting, human inter-rater reliability was Fleiss’ kappa = 0.307. These figures describe that study, not code review in general. Google Research, automated patch repair study (2026).
In practice, write down the rubric before enabling an automatic gate. Have reviewers refine it using examples where reasonable people disagree. Send ambiguous or high-impact findings to a person rather than forcing the model to choose a single label. Review disagreements and update both the rubric and the labeled test set deliberately.
More formal approaches to judge adaptation are also being studied. The 2026 FairJudge paper identifies task and domain adaptation, non-semantic biases, and evaluation inconsistency as limitations in existing LLM-as-a-judge systems, and proposes modeling judge behavior as an adaptive, regularized policy. This is a research proposal, not evidence of a universal fix for code review. PMLR, “FairJudge” (2026).
Outdated Drivers Are Slowing You Down
One free scan finds every outdated or missing driver and matches the right update for your exact hardware.Free scan · exact hardware matchPC Slower Than It Used to Be?
A free scan shows the junk files, broken settings and background clutter dragging Windows down - then fixes them in one click.Free scan · Windows 10 & 11When should an LLM review block a build?
An automatic gate is most defensible when the finding is clearly defined, the consequences of a false pass are understood, and the label-to-action mapping is explicit. Use a softer path when the rubric admits multiple reasonable interpretations or the cost of a false positive is high enough to disrupt work.
Quick Recap
- Use a hard failure for narrowly specified, high-impact categories when your team accepts the remaining classification risk and has a tested escalation path.
- Use a warning or human review for subjective quality judgments, uncertain labels, novel findings, malformed output, and cases where the critic’s rationale conflicts with its label.
- Use multiple judges cautiously. Diversity can help only if the judgments add meaningfully independent information; counting votes from highly correlated systems may create false confidence.
- Allow acceptable alternatives where appropriate. If a task has several defensible ratings, represent that ambiguity in the rubric and review process rather than forcing an arbitrary single answer.
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
