iTechGuides is reader-supported. When you buy through links on our site, we may earn an affiliate commission. As an Amazon Associate I earn from qualifying purchases. Learn more
Yes. A test can look redundant under one measure—such as statement coverage—yet still catch failures that other tests miss. But that does not mean every test should stay: the right decision depends on what the test checks, what the chosen measure leaves out, and the cost of keeping it.
What makes a test look redundant?
Test-suite minimization removes tests judged unnecessary under a chosen adequacy criterion, with the aim of reducing the effort of running the suite. For example, if two tests execute the same statements, a coverage-based tool may treat one as removable. That conclusion is limited to the criterion: identical statement coverage does not establish that the tests exercise the same states, boundaries, interactions, or failure behavior.
“Redundant” therefore means redundant according to a particular measure, not necessarily useless in every respect. Before removing a test, identify the exact reason it was flagged: duplicate input, shared requirement, statement or branch coverage, mutation score, or repeated behavior. Then check what that measure does not represent.
The Tool Desk
Outbyte PC Repair FREERepair Windows errors before they cause bigger problemsFix Now →Outbyte Driver Updater FREEScan for outdated or missing drivers - takes under a minuteDriver Scan →Why can a reduced suite miss failures?
A smaller suite may preserve its selected coverage measure while losing its ability to detect failures that matter. In a 2018 study of 1,478 failed builds from 32 GitHub projects, evaluated reductions missed up to 52.2% of failed builds under the study’s mappings. That is the maximum observed result in that study, not a general rate for software projects. The researchers also found that traditional reduction metrics did not predict the measured detection loss well. Read the ACM SIGSOFT ISSTA 2018 study.
#1 Best Overall
The practical lesson is to avoid treating a preserved coverage number as proof that a reduced suite is equally protective. A test may add value by checking a boundary, a distinct state, an interaction, or behavior associated with a past failure—even when another test appears to cover the same code.
How is minimization different from choosing or ordering tests?
These are related but distinct ways to manage regression-test effort:
- Minimization aims to make a smaller suite while retaining a chosen adequacy criterion.
- Selection chooses tests relevant to a particular change.
- Prioritization orders tests so that useful results, including failures, are more likely to appear earlier.
They solve different problems, so a test that is not needed in a minimized suite for one criterion may still be relevant to a particular change or worth running early. The distinction is covered in Yoo and Harman’s survey of regression testing techniques. Read the survey.
What does regression testing tell you?
Regression testing checks whether a change has caused failures in parts of a test item that were not modified. ISO/IEC/IEEE 29119-1:2022 defines it as “testing performed following modifications to a test item or to its operational environment, to identify whether failures in unmodified parts of the test item occur.” Retesting is different: it checks whether a correction fixed the targeted fault. The appropriate regression tests depend on the software item and the change. See ISO/IEC/IEEE 29119-1:2022.
How can mutation testing help evaluate your tests?
Mutation testing makes small artificial changes to code, then checks whether the tests detect them. A mutant that survives can point to a missing case or an assertion that does not check the behavior closely enough. It is a prompt to investigate, not proof that a real future bug will escape: a mutant can be equivalent to the original behavior, infeasible to expose, or too low-value to justify another test.
A 2021 Google Research study analyzed 15 million mutants and reported evidence that developers using mutation testing wrote more tests, and that mutants were coupled with real faults. Those findings support mutation testing as a way to find weaknesses; they do not establish that a particular future defect will be caught. Read the study record.
Use surviving mutants as targeted questions
- Review a surviving mutant and identify the behavior its change represents.
- Decide whether that behavior matters for the product and whether the mutant is meaningful and feasible to detect.
- If it exposes a real gap, add a focused case or assertion that checks the relevant behavior.
- Concentrate on risky and business-critical areas instead of pursuing a 100% mutation score as a universal target.
Microsoft’s .NET mutation-testing guidance recommends interpreting results rather than chasing a score indiscriminately. Read the .NET guidance.
Free tools Windows power users keep installed
One-click scans. No signup required.
When should you keep or remove a test?
Make the decision using evidence beyond whether a test is easy to remove:
Best Value
- Keep it when it covers a distinct state, boundary, interaction, or important failure mode that the retained tests do not clearly exercise.
- Consider removing or consolidating it when it repeats another test’s meaningful behavior and adds no useful detection, diagnostic clarity, or stability.
- Check project history where possible: assess whether the reduced suite would have detected real failures, rather than relying only on a proxy such as coverage.
- Account for maintenance cost: unstable tests and unclear diagnostics impose costs, but so does losing a useful warning.
- Triage mutation reports as well as tests. Google’s account describes limiting surfaced mutants to avoid spending reviewer attention on redundant or unhelpful results. Read Google’s account.
There is no universal rule to keep every test or delete every apparent duplicate. The useful question is whether a test contributes protection or diagnostic value that the remaining suite does not provide, weighed against its execution and maintenance cost.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
