iTechGuides is reader-supported. When you buy through links on our site, we may earn an affiliate commission. As an Amazon Associate I earn from qualifying purchases. Learn more
Because the application saved the answer to conversation history even though its check ended in an error, then sent that history to the model again on a later call. In a reported fix to OpenAI Agents JS, answers whose checks did not complete were withheld from that later context. Here, “remember” means replay through application-managed history—not model training or a chat product’s long-term memory.
What failed—and what “remembered” means
A check can end in two materially different ways: it can explicitly reject an answer, or it can throw an error before deciding. The reported code already withheld explicitly rejected answers. The defect was that an answer could remain available for replay when a check threw.
That does not mean the model learned the answer. The reported experiment examined application-managed conversation history: whether an answer was retained and then included in a subsequent model call. The authors state, “We did not test model training or a chat product’s long-term memory feature.”
Do these 3 things before closing this tab:
1Repair Windows errors before they cause bigger problems2Fix the driver behind crashes, sound loss and screen glitches3Clear out junk files and repair common Windows errorsWhat the reported fix changed
The fix treated an incomplete check as neither acceptance nor rejection. If checks did not complete, the answer was withheld from the next model call. Answers that passed remained eligible, as did previously accepted history. An error therefore stopped an unverified answer from being replayed as context; it did not establish that the answer was false.
#1 Best Overall
| Checker outcome | Answer replay under old logic | Answer replay under reported fix | Interpretation |
|---|---|---|---|
| All checks pass | 4/4 | 4/4 | Passing answers remained eligible for later context. |
| Explicit rejection | 0/4 | 0/4 | The old behavior already withheld rejected answers. |
| A check throws | 4/4 | 0/4 | The fix withheld answers when checking did not complete. |
| One check passes and another throws | 4/4 | 0/4 | A passing check did not make the answer eligible when another check failed to complete. |
These are constructed test-case counts reported by the experiment’s authors, not production rates or a general reliability measure. For example, 4/4 means the answer was replayed in all four tested combinations for that outcome and code version; it does not mean that 100% of real-world failures behave this way.
How the experiment was run
The authors report testing four checker outcomes across streaming and non-streaming runs, two session implementations, and two code versions: the old logic and the fix. That gives 32 observations: four outcomes × two run modes × two session types × two versions. They used a scripted model to supply distinctive answers while library code handled conversation and storage. The authors examined both stored items and the input to a later model call.
The report identifies OpenAI Agents JS pull request #1938, submitted by jbeckbeckwith-oai and merged on 2026-09-15 UTC. It gives candidate revision 457dfff8ce30d19ccbd4a3796ec482356dc19dfa; the control used the predecessor of runner/guardrails.ts from 8ac97dfedd0395beeb38edb0a17b83a9b89c3354. According to the report, runner/guardrails.ts was the pull request’s only changed production module.
The authors say they used the public run, ScriptedModel, and MemorySession APIs, plus an append-only session. They report no live model service or external database. They also report 71 passing tests in the corresponding upstream test file, including additional tool-output behavior; that count is separate from the 32 constructed observations and is not a reliability score.
This scope matters: the report does not establish correctness for every custom session store, nor does it establish that the same defect exists in CodeFlowMu. Its findings describe the tested code and setup, rather than a universal rule about AI systems.
What a failed check does—and does not—tell you
An unfinished check means the answer was not verified by that check; it does not prove the answer wrong. The authors put it plainly: “An unfinished check has not proved the answer wrong.” The reported fix is an admission rule for future model context, not a verdict on the answer’s truth.
Rank #4
Even a completed check is only as meaningful as its configured requirements. The authors caution: “Even completed checks establish acceptance only under their configured requirements. They do not guarantee real-world truth.” A passing result means the answer met the checks that ran, not that it is universally accurate.
Outdated Drivers Are Slowing You Down
One free scan finds every outdated or missing driver and matches the right update for your exact hardware.Free scan · exact hardware matchWindows Errors? Fix Them Before They Spread
Repair common Windows errors and clear accumulated junk for a smoother, more stable PC - no reinstall needed.Free scan · no reinstallHow to debug an answer that returns after an error
When investigating this pattern, trace the answer through storage and prompt assembly rather than assuming the model retained it independently.
Best Value
- Identify the checker outcome. Distinguish an explicit rejection from a thrown error or other incomplete check. They can lead to different history decisions.
- Inspect what the application stored. Check whether the answer remains in the session or conversation record after the error.
- Inspect the next model input. Determine whether the stored answer was actually included in the following call. Stored data is not necessarily sent as context.
- Check the admission rule. Verify that only answers meeting the intended acceptance criteria enter future model prompts, including when several checks run and one fails.
A useful debugging question is: “does an answer marked as failed later reappear as an already-established fact?” If so, the key distinction is whether “failed” means rejected, or merely that verification did not finish.
Keep diagnostic records separate from trusted context
A useful design implication is to retain failed attempts for debugging without automatically admitting them to future model prompts. These are separate purposes: a record can help investigate what happened while remaining ineligible as trusted context.
If verification happens later, the eventual verdict should be tied to the precise answer, turn, and checking rules it evaluates. Otherwise, a delayed result could be applied to different content or a different standard. The report raises this as a follow-up design question; it does not say that deferred verification or such an association mechanism was implemented or tested.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
