iTechGuides is reader-supported. When you buy through links on our site, we may earn an affiliate commission. As an Amazon Associate I earn from qualifying purchases. Learn more
A project report from Don Johnson describes putting Jev, a probability-returning model, behind a TLA+-specified consensus protocol and testing it in 1,680 simulated pharmacy decisions. The reported result was zero wrong verdicts, with human escalation increasing when severe failures were injected. That is evidence about this protocol in a finite synthetic test suite—not proof that Jev is clinically safe or suitable for pharmacy practice.
What the project built
Johnson’s account describes a practical problem: repeated requests to Jev could return slightly different probabilities, so checking for one exact output value was unsuitable. The proposed solution was to place a decision protocol around the model rather than rely on a single response.
According to the project repository, five agents submit validated paraphrases of a question. A stability gate checks whether their votes are sufficiently consistent, and the protocol requires a quorum of three stable votes out of five before reaching a verdict. Otherwise, it can escalate to a human.
Recommended Free Tools
The repository describes four TLA+ modules and model checking across 24 configurations, an AsyncAPI contract with fields traced to specification variables, generated Rust types, a Rust kernel, and a seeded pharmacy simulation. It also lists replayable TLC counterexample traces and 48 Rust tests. These are implementation details reported by the project, not independently inspected or verified here.
#1 Best Overall
What the 1,680 simulated decisions showed
The repository reports 360 rounds at each of three chaos levels. “Correct” means the simulation reached the expected verdict; “escalated” means it did not decide and referred the case for human review. The rounds used synthetic pharmacy scenarios.
| Chaos level | Rounds | Correct decisions | Escalated | Wrong verdicts |
|---|---|---|---|---|
| None | 360 | 360 | 0 | 0 |
| Realistic | 360 | 360 | 0 | 0 |
| Severe | 360 | 314 | 46 | 0 |
| Total | 1,080 | 1,034 | 46 | 0 |
The 1,680 figure covers the full suite, while the table’s 1,080 rounds are the “golden” rounds used for the reported zero-wrong-verdict result. Under the rule of three, Johnson gives a 95% confidence upper bound below 0.28% for the error rate in those golden rounds, assuming the method’s sampling conditions. Zero observed errors does not mean the true error rate is zero, and the bound is not a claim about real-world clinical decisions.
Under severe chaos, the reported escalation rate rose from 5.0% to 18.0% (z = 6.83). The intended behavior is important: when evidence or system reliability worsens, the protocol should be more likely to defer rather than force a decision.
The Tool Desk
Outbyte Driver Updater FREEScan for outdated or missing drivers - takes under a minuteDriver Scan →Outbyte PC Repair FREERepair Windows errors before they cause bigger problemsFix Now →How the system was stressed and how it failed safely
The reported chaos conditions included adversarial text inserted into patient records, truncated inputs, agent crashes, rate limits, and transport errors. An initial fail-fast response to transport errors reportedly voided 71% of severe-chaos rounds. The revised behavior marked lost agents unavailable and continued when the remaining responses allowed it; the repository reports no aborted rounds across the 1,680-round suite.
One example involved a documented penicillin anaphylaxis and a new amoxicillin order. With failures injected, the quorum was not met, so the system escalated to a human instead of returning a verdict. This illustrates the protocol’s fallback behavior; it does not establish that the model correctly handles medication allergies in practice.
What the model measurements add—and what they do not
Johnson reports capturing 1,490 calls to Jev version 1.13.0, with each response stored verbatim alongside a SHA-256 hash. The reported score spread was 0.042 for identical requests, 0.059 when question order changed, and 0.073 across a paraphrase cohort. These results motivate tolerances and stability checks instead of exact-value matching.
For 240 constructed items, the author reports accuracy of 0.979 and a Brier score of 0.0187. Reported latency was 96.7 ms for one question and 98.0 ms for 38 questions. Billing-meter behavior was reported linear to within one token across a 2,500× range. These are project measurements, not independently replicated benchmarks; the published account does not establish broader operating conditions for the latency figures.
Free tools Windows power users keep installed
One-click scans. No signup required.
The account says identical prompts across five agents stayed within the measured noise floor, while paraphrases produced more spread on difficult cases. Because all five agents use the same underlying model, they are not five independent judgments. Shared model errors could therefore pass through the quorum together.
Why a stable vote can still be wrong
A stability gate asks whether scores cluster. It does not establish that the record contains enough information to answer. The repository reports that an underdetermined scenario was escalated 86 times out of 120 and decided the other 34 times, split between yes and no. It also notes that a case labeled ambiguous was actually answerable, revealing a mistake in the scenario labels.
Rank #4
Those findings expose two distinct risks: a protocol can mistake uncertainty for a sufficiently stable signal, and the test’s own expected answers can be mislabeled. As Johnson puts it, “The stability gate catches jitter around a value. It cannot detect that no value is warranted.”
Independent reader supportYour contribution helps us test, update, and keep practical guides available for everyone.What TLA+ verification does—and does not—establish
Model checking can test whether a specified protocol satisfies modeled properties across explored configurations and assumptions. In this project, that supports claims about the consensus protocol’s behavior as specified. It does not validate Jev’s pharmaceutical knowledge, the correctness of scenario labels, the clinical appropriateness of a verdict, or the safety of a real deployment.
The repository explicitly says the pharmacy scenarios are synthetic, written to have unambiguous answers for protocol testing, and are not clinical guidance or formulary validation. Johnson’s statement is direct: “The pharmacy scenarios are synthetic, written to have unambiguous answers so the harness can detect protocol failures. Nothing here is clinical guidance.”
How to interpret the result
The most defensible conclusion is narrow: in the author’s reported synthetic test suite, the protocol produced no wrong verdicts in 1,080 golden rounds and escalated more often under severe chaos. That is a useful demonstration of a fail-closed design goal—uncertain or degraded operation can lead to review instead of a forced answer.
It is not evidence that the model or protocol is ready for pharmacy use. That would require clinical and formulary validation, trustworthy labels, and evidence that relevant real-world failure modes are handled; none is established by these project results. Voting among paraphrases may help expose output instability, but it cannot remove shared-model failure or determine answerability on its own.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
