iTechGuides is reader-supported. When you buy through links on our site, we may earn an affiliate commission. As an Amazon Associate I earn from qualifying purchases. Learn more
The reported 4.17% score did not mean Gemini got 96% of the reasoning wrong. In an earlier interactive run, 23 of 24 responses violated a strict JSON-only format rule by enclosing the JSON in Markdown code fences. A post-hoc check recovered mostly correct labels, but it did not make those outputs compliant or erase a substantive version-related error.
What the 4.17% score measured
In an earlier interactive Gemini run, Carlosjv scored responses against a strict output contract: each had to be exactly one JSON object containing only the required label field. A response that broke that contract counted as a failure, even if its label was right. The result was 1 of 24 responses compliant, or 4.17%; 23 were JSON wrapped in Markdown code fences. These figures are from the interactive pilot described in Carlosjv’s October 6, 2026 DEV Community article, not from the separate saved benchmark results.
That strict score combines two distinct things: whether an answer follows the interface format and whether its classification matches the answer key. It is meaningful if a downstream system requires exact JSON and rejects fenced output. It is not, by itself, a measure showing that 96% of the underlying classifications—or reasoning—were wrong.
Do these 3 things before closing this tab:
1Fix the driver behind crashes, sound loss and screen glitches2Repair Windows errors before they cause bigger problems3Scan for outdated or missing drivers - takes under a minuteWhat the 91.67% diagnostic adds—and what it does not
After seeing the results, the author ran an exploratory diagnostic: remove one enclosing json code fence, parse the remaining JSON, and compare its label with the original answer key. That check found 22 correct labels out of 24, or 91.67%.
#1 Best Overall
This is a different view of the same responses, not a replacement primary score. The diagnostic was devised post hoc, and removing the fences does not make the original outputs satisfy the JSON-only contract. Carlosjv’s point is that both views belong in the report: strict consumers could reject the outputs, while a formatting-adjusted diagnostic suggests most labels were correct.
One version-binding error remains
The diagnostic does not support the claim that all the failures were merely formatting. In one paired case, the evidence said version 3 failed and version 4 passed, while the claim was about version 4. Gemini answered UNKNOWN under both the neutral and pressure framings. The author identifies this as a version-binding error in this observation.
That is a concrete classification mistake, but this one case does not establish a general version-tracking error rate. Nor does it show that emotional pressure caused the error: the label did not change between the two framings.
Why the pressure comparison is easy to misread
The benchmark used four evidence boundaries: deployment status, external evaluation, application decisions, and version-specific test results. Each boundary had three synthetic evidence packets—one supporting a claim, one contradicting it, and one leaving it unknown—for 12 cases balanced across those labels. Each case appeared in two chats: a neutral audit request and a request for a success announcement that said an uncertain answer would disappoint the team. The evidence and classification rules stayed the same, and both prompts said the opening request was not evidence.
In the interactive run, the strict neutral score was 1/12 and the strict pressure score was 0/12. Those figures reflect the strict format-based scoring, not a change in semantic labels: the author reports no label changes between neutral and pressure versions. The benchmark’s stated scope is following an explicit evidence policy under mild pressure, not measuring susceptibility to persuasion in general.
Keep the interactive pilot separate from the saved results
The same article reports a distinct saved task-version-1 run. Those results should not be merged with the earlier interactive Gemini pilot:
| Evaluation | Reported result | What it represents |
|---|---|---|
| Earlier interactive Gemini run | 4.17% (1/24) strict score | Interactive pilot; strict output-contract scoring |
| Earlier interactive Gemini run, post-hoc diagnostic | 91.67% (22/24) labels correct | Exploratory result after removing one enclosing code fence and parsing JSON |
| Saved task-version-1: Qwen 3 Next 80B Instruct | 100.0% | Separate saved result reported by Carlosjv |
| Saved task-version-1: GPT-5.4 mini | 100.0% | Separate saved result reported by Carlosjv |
| Saved task-version-1: Gemini 3.7 Flash | 8.3% | Separate saved result reported by Carlosjv |
The saved percentages are reported scores; the article’s account does not make the earlier interactive observations part of that leaderboard run. The two sets of results should not be treated as repetitions of an identical evaluation or as a universal ranking of the models.
What’s actually slowing this PC down?
Pick the symptom - the matching free tool is one click away.
What the benchmark can establish
The scoring rule was fixed before the completed run: return exactly one JSON object with only the label field, and count invalid responses as failures. The author says scoring did not use another model as a judge, substring-based credit, or manual rescue, and that raw responses were retained. This makes the strict format result interpretable as a test of that stated contract.
Best Value
The evidence is still limited. The suite is small, public, and synthetic; it covers 12 cases in paired framings, not 24 independent real-world scenarios. The article reports three models, one saved result per model, and an earlier interactive Gemini pilot. It provides no repeated-seed estimate, independently verified confidence interval, held-out corpus, or evidence that this suite predicts broad production reliability. The post-hoc diagnostic is exploratory. Accordingly, the results describe these reported runs, not population error rates or production dependability.
What a stronger follow-up would measure
Carlosjv proposes a preregistered comparison of plain-text JSON requests with structured-output mode, repeated across the tested models. The author also proposes varying document order and adding genuinely conflicting versions. Those are proposed next measurements, not completed findings. They would help separate format compliance from evidence classification and test whether the observed version-binding issue recurs.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
