iTechGuides is reader-supported. When you buy through links on our site, we may earn an affiliate commission. As an Amazon Associate I earn from qualifying purchases. Learn more
Flaky tests persist partly because fixing one can be uncertain, time-consuming work competing with changes whose value is easier to see. That makes deferring a difficult investigation understandable at the individual-team level—but costly when reruns, noisy signals, and reduced confidence accumulate. The incentive explanation is a plausible interpretation of documented engineering constraints, not proof that every team ignores flaky tests or that incentives alone cause them to persist.
What makes a test flaky—and why its failure matters
A flaky test can pass and fail against the same code under different runs. Its result is therefore a noisy signal: a failure may reflect a test problem or an unstable environment, but it may also reveal a real production-code fault. Microsoft’s industrial study warns that ignoring flaky-test failures can hide such faults (Microsoft Research, 2019).
There is no single root cause. Reported sources include asynchronous work, timing, concurrency, nondeterministic or undefined behavior, third-party dependencies, and variation in infrastructure, hardware, or environment. A 2023 literature review also identifies nondeterminism in machine-learning applications; those categories describe possible causes, not a diagnosis or prevalence estimate for any particular suite.
Recommended Free Tools
Why a team may defer the investigation
The immediate cause of an intermittent failure is often uncertain. Reproducing it and tracing its origin can take substantial effort, especially when it does not appear consistently or is unrelated to the most recent change. Google’s study of Flake Aware Culprit Finding reports more than 13,000 test breakages and describes the cost of reruns needed to attribute failures (Google Research, 2023).
That creates a practical trade-off. An engineer can spend time on a difficult failure with unclear cause, or on a product change with a clearer short-term outcome. Under schedule pressure, deferral can be locally rational even if nobody wants a noisy test suite. This decision model is an inference from the documented effort and resource constraints; the studies do not establish that all teams make this choice or measure how much it causes flakiness to persist.
The consequences can spread beyond one test. Reruns consume CI and engineering time; ambiguous failures complicate attribution; repeated noise can weaken confidence in the suite. Mozilla’s study of 21 professional developers and 200 previously fixed flaky tests reports that developers perceived effects on resource allocation, scheduling, and test-suite reliability (Mozilla Foundation, 2019).
Why “just rerun it” is not a fix
A passing rerun shows that the test passed on that attempt; it does not show that the underlying cause has been removed. In a lifecycle study of six proprietary Microsoft projects, researchers found cases where developers said they had fixed a flaky test but empirical experiments did not find a reduced frequency of flaky failures (Microsoft Research, 2020).
That same study found asynchronous calls to be the leading cause in those six projects. The result is specific to that sample, not a universal ranking. It also evaluated FaTB on five flaky tests: the tool reduced running time by up to 78% without empirically affecting the frequency of flaky failures. That is a runtime result, not evidence that the tests’ flakiness was fixed.
Reruns can still be useful as diagnostic evidence, particularly when a failure is intermittent. But a green rerun alone cannot distinguish a transient environment issue from an unresolved test defect or a genuine regression.
Independent reader supportYour contribution helps us test, update, and keep practical guides available for everyone.How to respond without hiding a real failure
A useful response makes uncertainty visible while preserving a route to diagnosis. Google’s practitioner guidance identifies concurrency, nondeterministic or undefined behavior, third-party code, and infrastructure problems among causes teams may need to investigate (Google Testing Blog, 2016). Microsoft’s report that more than 100 product teams used its flaky-test management system illustrates that managing these failures can be an ongoing organizational activity, rather than a one-time cleanup (Engineering at Microsoft, 2022).
Rank #4
- Preserve the failure signal. Record the failing test, run context, and whether a rerun changed the result. Do not treat a pass on retry as proof of repair.
- Seek evidence about the cause. Check for asynchronous work, timing assumptions, shared state or concurrency, external dependencies, and environmental variation. These are investigation paths, not a presumption that any one category is responsible.
- Verify a proposed fix. Look for a reproducible reduction in failures or diagnostic evidence that addresses the suspected cause, rather than relying only on the claim that the test was fixed.
- Make deferral explicit and owned. If immediate repair is not feasible, keep a clear path for follow-up so that a flaky result does not become an indefinitely ignored signal.
The key distinction is between managing uncertainty and discarding evidence. A team may need to prioritize work, but an unowned failure that repeatedly disappears on rerun can continue to obscure both test defects and real faults.
Do these 3 things before closing this tab:
1Fix the driver behind crashes, sound loss and screen glitches2Repair Windows errors before they cause bigger problems3Scan for outdated or missing drivers - takes under a minuteQuick Recap
Best Value
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
