iTechGuides is reader-supported. When you buy through links on our site, we may earn an affiliate commission. As an Amazon Associate I earn from qualifying purchases. Learn more
A flaky test that fails and then passes on rerun has produced a new result—not proof that the test or the code is reliable. Treat the green retry as a clue to investigate, not a repair. The specific incident suggested by this title is not established; the guidance below applies whenever a test passes sometimes and fails sometimes without a noticeable change.
What a flaky test tells you
Martin Fowler defines a non-deterministic test as one that “passes sometimes and fails sometimes, without any noticeable change in the code, tests, or environment.” (Martin Fowler, “Eradicating Non-Determinism in Tests,” 14 April 2011.) A failure followed by a pass is consistent with that pattern, but by itself it does not identify the cause.
The practical problem is lost signal: when a regression test fails intermittently, it is harder to tell a real defect from nondeterminism. If the team learns to dismiss such failures, confidence in the suite can erode. Retrying until green conceals that uncertainty rather than explaining it.
How retries, quarantine, and fixes differ
| Action | Restores a trustworthy regression signal? | What it does to the cause | Feedback delay | Ownership and repair expectation |
|---|---|---|---|---|
| Retry | No. A pass is another observation, not evidence that the test is reliable. | May help reveal intermittency, but does not explain or remove its cause. | Adds another test run; the amount depends on the setup. | Not stated in Fowler’s guidance. |
| Quarantine | No. It contains the unreliable test rather than restoring its signal. | Separates the test from ordinary runs while the cause remains to be fixed. | Can reduce disruption to normal test feedback; a specific delay is not stated. | Fowler advises fixing quarantined tests quickly; assign an owner and prompt repair expectation. |
| Fix | Yes, when the underlying source of nondeterminism is removed and the test is reliable again. | Addresses the cause rather than masking it. | Requires investigation and repair; no duration is specified. | Make responsibility for the repair explicit. |
Fowler’s containment advice is: “Place any non-deterministic test in a quarantined area. (But fix quarantined tests quickly.)” Quarantine is temporary containment, not a destination for tests that no one owns. The article does not prescribe a retry count or a CI-vendor setting.
Investigate the conditions that change the result
Fowler names recurring sources to check. These are diagnostic leads, not proof that any one is responsible in your project:
- Shared state or poor isolation: one test may leave data or process-wide state that changes another test’s result.
- Asynchronous work checked with a fixed sleep: a guessed delay can be too short under some conditions and unnecessarily long under others.
- Remote-service dependence: external availability or behavior can affect a test that depends directly on a service.
- Direct use of the system clock: time-dependent behavior can vary as the clock advances or around timing boundaries.
- Resource leaks: unreleased resources can make failures appear later or migrate between tests.
A repair-oriented workflow
- Record and reproduce the failure. Capture the failing test and relevant conditions. Separate code changes from variation in state, timing, machine load, and external dependencies.
- Control the starting state. Isolate tests so one cannot leave data or process-wide state that changes another’s result.
- Wait for behavior, not a guessed interval. For asynchronous work, wait for an observable condition with a bounded timeout instead of sleeping for a fixed duration.
- Make unstable dependencies controllable where appropriate. Use a test double for a remote service and validate its contract separately. Inject a controllable clock for time-dependent behavior.
- Inspect teardown and cleanup. Check resource release when failures migrate between tests or appear after a long run.
- Contain only when necessary. If immediate containment is needed, quarantine the test, name an owner, and set a prompt repair expectation. Remove the quarantine after the test is fixed.
Further reading
Fowler’s article recommends Gerard Meszaros’s xUnit Test Patterns for further reading. Current edition and availability are not established here.
Quick Recap
Best Value
Rank #4
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
What’s actually slowing this PC down?
Pick the symptom - the matching free tool is one click away.
