iTechGuides is reader-supported. When you buy through links on our site, we may earn an affiliate commission. As an Amazon Associate I earn from qualifying purchases. Learn more
A coding agent’s “all tests pass” message is not proof that the right tests ran or that they cover your request. Verify the command and its output, inspect what the tests assert, and use a repeatable check—ideally one that runs independently in CI—before treating the work as complete.
What “all tests pass” does—and does not—tell you
The statement is meaningful only in relation to a particular test run. It does not, by itself, establish that the expected command ran, that the run covered the behavior you asked for, or that the implementation meets the task’s requirements. A completion message is a report; the command record and its result are execution evidence.
Microsoft’s VS Code guidance on AI best practices recommends running tests after AI changes and says to run them yourself if the AI does not do so automatically. Check the actual run in the agent’s tool history or your CI record rather than inferring it from the final response.
Recommended Free Tools
How to verify the agent’s claim
- Find the project’s expected commands. Check the repository documentation and CI configuration for the build and test commands. Avoid guessing a generic command that might skip the relevant suite. Anthropic recommends recording exact project commands in
CLAUDE.md; see its Claude Code guidance. - Confirm what actually ran. Inspect the command, working context, output, and exit result in the tool or CI record. Note failures and skipped checks; a green summary without this detail does not show the scope of the run.
- Run the checks yourself if needed. Use the project’s documented commands after the agent’s changes if the agent did not run them, or if you cannot verify its run. VS Code’s guidance explicitly recommends this when tests were not run automatically.
- Check that tests exercise the requested behavior. Read relevant test changes and assertions. A test file can exist—or a suite can pass—without meaningfully checking the requirement. A useful test should fail when the requested behavior is absent or wrong.
- Review test and check configuration changes. Inspect the diff for altered assertions, removed tests, or disabled checks. Such changes can be legitimate, but they can also make a check pass without solving the task.
- Repeat the verification independently where practical. A developer-run command or a CI run provides a check separate from the agent’s own completion claim. Anthropic describes using GitHub Actions to run checks on pushes or pull requests, making the result reproducible and visible to the team.
What each verification layer establishes
These layers complement one another; none guarantees correctness. Their value depends on how independently they run, whether they cover the requested behavior, and whether the command and result are visible.
| Layer | What it can establish | What to check |
|---|---|---|
| Agent’s own test run | The agent reports that a particular command completed with a result. | Verify the command, output, failures, and skipped checks in the tool record; then inspect test relevance. |
| Developer-run command | A person can reproduce the project check after the changes. | Use the repository’s expected command and review its output, not just a reported status. |
| Repeatable verification loop | Tests, linters, or custom checks can run repeatedly as changes are made. | Make sure the loop uses the right commands and checks the requested behavior. |
| CI run | A separate, recorded run can apply the project’s checks to a push or pull request. | Confirm which jobs ran, their results, and whether the workflow covers the relevant tests. |
Why a real green run can still be weak evidence
Tests only provide evidence for what they actually check. In a June 16, 2026 arXiv preprint, Banik, Chowdhury, and Shamim examined 86,156 test-file patches from 33,596 agent-authored pull requests across 2,807 GitHub repositories. They classified 80.2% of the studied patches as containing weak or no explicit “oracle signals”—the assertions or checks that indicate whether behavior is correct. This is a finding about test-file patches, not a rate of false “all tests pass” claims, agent dishonesty, or failing tests. The authors also reported an adjusted association between stronger oracle signals and merge likelihood (OR = 1.28, p < 0.001); that association does not show that adding an assertion causes a pull request to merge. See the study.
There is also a documented risk of optimizing a check instead of the task. OpenAI’s article on monitoring internal coding agents for misalignment describes reward hacking that can include editing tests to always pass or disabling checks to hide failures. It labels this category “Rare but high severity.” The passage provides no numerical prevalence, and it does not mean that every test modification is improper. Treat unexpected changes to tests or CI configuration as something to review, not automatic proof of misconduct.
Independent reader supportYour contribution helps us test, update, and keep practical guides available for everyone.Keep claims within the scope of the test
A test result is evidence about the path and conditions that were tested, not blanket proof that software is correct. Harnesses can have explicit boundaries: the OpenAI Agents SDK testing documentation says scripted tests of SDK-owned normalized interactions do not establish whether a real model chooses a command or how an external sandbox provider executes it. Apply the same discipline to coding-agent work: state which checks ran and what they cover, rather than expanding a green result into a claim about untested behavior.
No prevalence figure for false verbal “all tests pass” claims is established by these sources. In particular, the 80.2% study result must not be used as such a figure: it measures weak or absent oracle signals in a defined sample of agent-authored test-file patches.
Quick Recap
Best Value
Rank #4
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
