iTechGuides is reader-supported. When you buy through links on our site, we may earn an affiliate commission. As an Amazon Associate I earn from qualifying purchases. Learn more
A flaky test can pass and fail on the same code. That means a red CI result is not, by itself, proof of a new regression—and repeated unexplained failures can make developers discount the test suite. A tool that surfaces suspicious tests can help restore signal, but identifying a likely flake is the start of diagnosis, not the fix.
What makes a test flaky?
John Micco’s 2016 account of Google’s experience defines a flaky result as one in which “the same test exhibits both a passing and a failing result with the same code.” Google Testing Blog Fuchsia’s policy uses a similar test: a test sometimes passes and sometimes fails when run using the exact same code revision. Fuchsia flaky test policy
This definition describes inconsistent behavior; it does not identify the cause. The test may be sensitive to shared state or execution order, but the runner, application, dependencies, or environment may also be responsible. A test that fails consistently because of a genuine defect is not flaky merely because the failure is inconvenient.
Why flaky tests erode CI confidence
Fuchsia’s policy warns that flaky tests can let real bugs pass through the commit queue, devalue otherwise useful tests, and increase queue failures and latency. Fuchsia flaky test policy When a test produces unexplained failures, developers face a costly choice: investigate every red result, or start treating failures as noise. Either response can weaken the value of CI.
Google’s 2016 figures illustrate the issue within Google’s own testing environment at that time—not an industry-wide or current rate. Micco reported that about 1.5% of test runs had a flaky result, almost 16% of tests had some level of flakiness, and about 84% of observed post-submit pass-to-fail transitions involved a flaky test. These figures describe different denominators and contexts, so they should not be combined or treated as directly comparable. Google Testing Blog
Where flakiness can come from
Google’s 2021 guidance groups potential causes into four parts of a test system. Google Testing Blog
- The test itself: setup, initialization, cleanup, or test-data assumptions may leave shared state behind or depend on another test’s behavior.
- The test-running framework: resource allocation or framework behavior may affect whether the test starts and runs reliably.
- The application and its dependencies: the system under test may not have started successfully, or an external dependency may behave unpredictably.
- The execution environment: the operating system, hardware, or network can introduce conditions the test does not control.
These categories are a triage map, not proof that any particular detection tool checks all four automatically.
Do these 3 things before closing this tab:
1Repair Windows errors before they cause bigger problems2Scan for outdated or missing drivers - takes under a minute3Clear out junk files and repair common Windows errorsHow to investigate a suspicious test
A useful flakiness report should help a developer move from “this test has inconsistent outcomes” to evidence about why. Work through the test and its environment rather than assuming the test code is at fault.
- Confirm the inconsistency. Compare runs of the same test against the same code revision and inspect whether outcomes differ. A failure on one revision followed by a pass after a code change does not, by itself, establish flakiness.
- Run the test independently. If it behaves differently alone than in the suite, investigate order dependence, shared state, and assumptions about previous tests.
- Review setup, cleanup, and test data. Check whether initialization is complete, cleanup leaves state behind, or tests reuse data or resources in ways that can collide.
- Inspect asynchronous behavior and timing. Look for races, timeouts, and events that the test assumes will happen within a fixed interval. Prefer synchronizing on an explicit application state over an arbitrary sleep; a delay can become unreliable again and makes the suite slower. Google Testing Blog
- Check the runner and system under test. Verify that required resources were allocated and that the application or service started successfully before the test made assertions.
- Examine uncontrolled dependencies and environment. Look for network, operating-system, hardware, or dependency conditions that varied between runs and were not isolated or accounted for.
- Turn the finding into a repair. Record the evidence and fix the reproducible cause where possible. A label or failure history can focus the investigation, but cannot substitute for it.
What a flaky-test tool can—and cannot—tell you
A tool can make unreliable tests easier to find by highlighting inconsistent outcomes and giving teams a place to focus investigation. The title alone does not establish a particular tool’s detection method, accuracy, performance, integrations, or ability to diagnose root causes. Those details need to be demonstrated for the specific tool; a list of suspicious tests should not be read as proof of why each one failed.
When evaluating a detection approach, ask what evidence supports each flag and whether it helps reproduce or narrow down the cause. Also weigh its confidence, added runtime and compute cost, risk of hiding a genuine regression, fit with the team’s CI workflow, and usefulness for root-cause investigation. These are practical tradeoffs, not a standardized score.
Rank #4
Retries and quarantine are containment, not repair
Automatically retrying a failed test can reduce false alarms, but requiring repeated failures before reporting one may delay discovery of a genuine regression. Micco describes both the value and the tradeoff of retry-based mitigation. Google Testing Blog
What’s actually slowing this PC down?
Pick the symptom - the matching free tool is one click away.
Quarantine can remove a highly flaky test from the critical path so it stops disrupting routine changes. But if the team then stops investigating it, quarantine can hide a real race or bug. Fuchsia’s policy is explicit: remove flakes from the critical path quickly, but do not ignore them after removal. Fuchsia flaky test policy
Best Value
Keep quarantined tests visible, assign follow-up ownership, and track whether the underlying issue is resolved. If a test is no longer blocking CI, it still needs a route back to reliable coverage rather than becoming a permanent blind spot.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
