iTechGuides is reader-supported. When you buy through links on our site, we may earn an affiliate commission. As an Amazon Associate I earn from qualifying purchases. Learn more
A green exit status means a command finished without reporting failure according to that command’s rules. It does not, by itself, prove that tests ran, that the intended tests were selected, or that the run produced evidence relevant to the question your check is meant to answer.
What an exit code does—and does not—tell you
An exit status is a process’s reported outcome, interpreted according to the command, runner, wrapper, and configuration that produced it. A zero commonly means the command did not report failure; Seth Wheeler, writing about his didrun project, summarizes that as “I did not fail.” That is a useful framing, not a universal formal definition of zero.
For a test check, separate four questions:
- Did the process run? Did the command start and complete, rather than being skipped, interrupted, or timed out?
- Did it report failure? What does this runner’s exit status mean under the effective configuration?
- Did the intended work happen? Were tests discovered, selected, and executed in the expected scope?
- Did the result answer the intended question? Does the run provide evidence that would catch the defect or condition the check is supposed to detect?
A positive answer to one question does not settle the others. Even a runner that distinguishes “no tests” from “tests passed” cannot establish that the right tests were chosen or that the test plan was adequate.
The Tool Desk
Outbyte Driver Updater FREEScan for outdated or missing drivers - takes under a minuteDriver Scan →Outbyte PC Repair FREEClear out junk files and repair common Windows errorsFree Scan →How test runners handle an empty test selection
There is no single no-tests rule for all test runners. These documented examples differ, and each result depends on the relevant runner’s behavior and configuration.
| Runner | Documented no-tests behavior | Configuration or scope to check |
|---|---|---|
| pytest | Exit code 0 means all tests were collected and passed; exit code 5 means no tests were collected. | Confirm that the intended tests were selected. The documented distinction does not prove that the test scope or plan was adequate. |
| Vitest | passWithNoTests defaults to false; enabling it allows Vitest not to fail when no tests are found. |
Review the effective configuration and command line, including whether --passWithNoTests is set. |
| Microsoft vstest | A filter matching no tests, or no discovered tests, produces a warning and does not fail by default. | RunConfiguration.TreatNoTestsAsError can make a zero-test run return 1. Check the filter and effective run configuration. |
These are runner-specific behaviors, not rules that can be generalized across frameworks. Versions, wrappers, plugins, command-line options, and configuration may change what a green result means.
Collection is not execution, and execution is not proof of coverage
“Tests collected” means a runner found tests under its discovery and selection rules. It does not automatically mean every relevant test executed or that the intended scope was included. Skips, filters, markers, environment conditions, and setup failures can all affect what a run actually exercised.
Wheeler’s article uses an all-skipped pytest run to illustrate the gap between collection and execution. The pytest exit-code reference establishes the distinct no-tests-collected code, but does not independently establish that all-skipped example. Treat the example as Wheeler’s illustration, not as a conclusion drawn from the exit-code page.
Do these 3 things before closing this tab:
1Repair Windows errors before they cause bigger problems2Scan for outdated or missing drivers - takes under a minute3Clear out junk files and repair common Windows errorsA text check can also be too weak. Wheeler describes a wrapper that could accept output containing “passed” even when the run had zero tests. A passing word is evidence only if the output also establishes a meaningful count and the relevant outcome. His article reports a project-specific result in which didrun caught six intentionally introduced mutations; that is an author-reported example, not an independently verified benchmark or industry statistic.
Rank #3
What to inspect in a green CI run
Wheeler’s article recommends checking positive counts, expected output, and whether an artifact was written or changed during the current run. These are practical evidence checks, not independently tested guarantees about every runner or wrapper.
- Inspect counts: Find the number of tests or work items actually executed, not just discovered. Where appropriate, set a meaningful minimum rather than accepting zero.
- Verify scope: Check the selected files, suites, filters, markers, packages, or targets against what the job is meant to cover.
- Require relevant output: Prefer structured runner output or an explicit count over matching a generic word such as “passed.” Make sure a success message cannot coexist with an empty run.
- Check artifacts against this invocation: Confirm the report or result file was created or updated during the current run. An old file already present on disk is not proof that this invocation produced it.
- Classify failure types: Distinguish an expected test failure from a syntax error, setup problem, or infrastructure failure. A check intended to catch a particular regression should not treat an unrelated failure as proof that it worked.
- Treat incomplete runs as incomplete: A timeout, interruption, or cancellation should not be counted as a successful completed check merely because it did not produce a conventional test failure.
Use evidence predicates carefully
Wheeler describes didrun as classifying outcomes such as ran-and-passed, ran-and-failed, did-not-run, and ran-and-failed-wrongly. The project’s approach requires at least one declared evidence predicate; examples in the article include matching output, parsing a count with a minimum, observing a file written during the run, and requiring a minimum duration. Wheeler describes duration as weak evidence and prefers a count. These are descriptions of didrun, not an endorsement or independent evaluation of the tool.
The useful principle is broader than any one wrapper: state what evidence would demonstrate that the check ran and did meaningful work, then make the CI job enforce it. A duration threshold alone cannot show that the right tests executed; a count can show activity but not necessarily relevant coverage. Combine evidence that addresses both activity and scope where the risk warrants it.
What’s actually slowing this PC down?
Pick the symptom - the matching free tool is one click away.
Test the guard, not only the tests
A CI check can itself be wrong while reporting green. Validate the guard with controlled cases: an empty selection should be rejected if emptiness is invalid; a known relevant failure should be detected; and an unrelated setup or syntax failure should be classified distinctly where the workflow depends on that distinction. This verifies the status logic rather than assuming the status proves it.
Best Value
Wheeler reports that his example go test ./... printed [no test files] while exiting 0, and that a wrapper invocation returned 3. Those are examples reported in his article and tool README description; the sources cited here do not independently verify them against official Go documentation. Do not treat them as a universal statement about Go behavior across versions or configurations.
The operational lesson is to read a green status together with the run’s evidence: what was selected, what actually executed, and whether the result could answer the question the check is meant to answer.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
