Quick wins for a faster PC:
Fix the driver behind crashes, sound loss and screen glitchesFind Drivers →Repair Windows errors before they cause bigger problemsFix Now →iTechGuides is reader-supported. When you buy through links on our site, we may earn an affiliate commission. As an Amazon Associate I earn from qualifying purchases. Learn more
A passing test suite shows that the checks it ran passed for the inputs and environment they covered. It does not prove the software is correct: relevant cases may be missing, or the tests may run the faulty code without asserting an outcome that exposes the fault. Coverage can reveal code that never ran; mutation testing can help assess whether tests would catch plausible mistakes in code that did run.
What a passing test run actually tells you
A green result is evidence about a bounded set of checks, not a guarantee about every behavior the software could exhibit. A test can only reject a defect if it reaches the relevant behavior and its assertions would fail when that behavior is wrong.
For example, a test might call a calculation function but only assert that it returns a number. If a bug changes the result from the correct value to a different number, the test can still pass. Similarly, tests may pass for ordinary inputs while missing boundary values, error conditions, or interactions with other components.
Why coverage does not prove test quality
Code coverage helps identify whether tests executed particular lines or branches. That is useful for finding code that received no test execution, but execution alone does not show that the tests checked the right result. Google’s discussion of coverage and mutation testing explains this distinction: Google Testing Blog: Code coverage best practices.
#1 Best Overall
A line can be covered by a test whose assertions are too weak to detect a wrong value. Coverage is therefore a map of where tests went, not a verdict on whether they would catch defects there.
How mutation testing probes whether tests can catch mistakes
Mutation testing makes controlled, small changes to code—such as changing a condition or an operator—and then runs the tests. If a test fails, it has detected that change, or “killed” the mutant. If the tests still pass, the mutant survived, which can indicate that the test suite did not detect a plausible fault in that area.
Google describes applying mutation testing to code changes during review, so surviving mutants can help reviewers spot test gaps: Google Testing Blog: Mutation testing. A survivor is a prompt to investigate, not automatic proof of a missing test: some mutations may be redundant or have no meaningful effect. Likewise, a mutation score is not a correctness guarantee.
Empirical evidence supports mutation testing as a useful signal, with important limits. A 2021 study record reports analysis of 15 million mutants and evidence that developers using mutation testing wrote more tests; the study also found mutants coupled to real faults in its dataset. Those findings describe the study’s scope, not a guarantee that mutation testing eliminates defects: Google Research: The practice and research of mutation testing.
How flaky tests weaken a green signal
A flaky test passes and fails on the same code. When results vary without a corresponding code change, a green run is harder to interpret: it may reflect inconsistency rather than a meaningful improvement or regression.
In a 2016 account, Google’s John Micco reported that about 1.5% of test runs were flaky, about 16% of tests had some level of flakiness, and about 84% of observed transitions from pass to fail involved a flaky test. These are historical figures from Google’s test corpus, not current or industry-wide estimates: Google Testing Blog: Flaky Tests at Google and How We Mitigate Them.
Independent reader supportYour contribution helps us test, update, and keep practical guides available for everyone.Build a testing strategy around risk
There is no universal number of tests or coverage percentage that makes a release safe. The right mix depends on the software and the people who rely on it. Google recommends using multiple testing layers, including unit tests, integration tests, end-to-end tests for critical user journeys, and other relevant tiers: Google Testing Blog: Just say no to more end-to-end tests.
Do these 3 things before closing this tab:
1Repair Windows errors before they cause bigger problems2Fix the driver behind crashes, sound loss and screen glitches3Clear out junk files and repair common Windows errorsQuick Recap
Best Value
- Unit tests check focused behavior, including boundary cases and error handling.
- Integration tests check interactions between components where defects can arise at their boundaries.
- End-to-end tests exercise critical user journeys through the system, where failures would affect users directly.
- Coverage checks help locate code that tests do not execute.
- Mutation testing can probe whether assertions detect plausible faults in exercised code.
- Flakiness monitoring helps distinguish dependable test results from inconsistent ones.
What to investigate when the suite is green but a defect remains
- Reproduce the defect. Identify the input, user journey, or system condition that produces the wrong behavior.
- Find the relevant check. Determine whether an existing test reaches that behavior and whether its assertions would reject the incorrect result.
- Use coverage as a locator. Check whether the affected code executes under tests, while remembering that execution does not establish assertion quality.
- Consider a targeted mutation. Ask whether a plausible small change to the affected logic would make a test fail. A surviving mutation can help expose a gap.
- Check result stability. If tests pass and fail on unchanged code, address the flakiness before relying on their status as release evidence.
- Add coverage at the right layer. Choose a focused unit, integration, or end-to-end test based on where the failure occurs and how users encounter it.
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
