iTechGuides is reader-supported. When you buy through links on our site, we may earn an affiliate commission. As an Amazon Associate I earn from qualifying purchases. Learn more
A false positive can make a conversion experiment look like a winner when the change has no real effect. A 5% significance level limits the chance of that result for one properly conducted test when there is truly no difference; it does not mean a reported winner has a 95% chance of being real, or that exactly 5% of declared winners will be false. The share of false discoveries among reported winners also depends on how many tested changes genuinely work and how results are selected.
What a false positive means in an A/B test
A false positive, also called a Type I error, occurs when a test reports a difference even though the null hypothesis of no true difference is correct. In a conversion experiment, that can lead a team to deploy a change that appeared to improve its chosen metric but does not actually outperform the control.
The significance level, often written as alpha, sets the tolerated Type I error rate for an individual test under the test’s statistical assumptions. Adobe describes the choice as a trade-off: a lower tolerance for false positives can make it harder to detect a real difference, increasing the chance of a Type II error, or false negative. The threshold alone does not give the probability that a specific reported winner is genuine.
Quick wins for a faster PC:
Scan for outdated or missing drivers - takes under a minuteDriver Scan →Repair Windows errors before they cause bigger problemsFix Now →Fix the driver behind crashes, sound loss and screen glitchesFind Drivers →That distinction matters because teams usually care about the proportion of declared winners that are ineffective, not only the error rate of one test under a null hypothesis. This proportion is the false discovery rate. It depends partly on how common truly effective changes are among the ideas being tested, as well as on test design and how many results are examined or selected.
#1 Best Overall
How often do significant results fail to hold up?
In a 2022 study of commercial website experiments, Ron Berman and Christophe Van den Bulte analyzed 4,964 effects from 2,766 experiments. They estimated that about 70% of effects were true nulls. In that study context, the estimated false discovery rate was 18–25% among results significant at 5%, and 28–37% among results significant at 10%. The authors summarized the 5% finding as an implication that about one in five significant interventions may be ineffective in the field.
Those figures are estimates from the study’s commercial experimentation data, not a universal rate for every company or testing program. They illustrate why a 5% per-test significance threshold and the false discovery rate among selected winners are different quantities: when many tested ideas have no real effect, a substantial share of the significant results can still be false discoveries.
Why repeated comparisons raise the risk
Each additional comparison creates another opportunity to find a result that looks significant by chance. Risk can accumulate when a team tests many variants, compares multiple offers, or searches segments after seeing the overall results. Adobe’s guidance specifically cautions against post-test segmentation and testing multiple offers without accounting for the comparisons.
Repeatedly checking results and stopping as soon as a p-value crosses the chosen threshold also changes the test procedure. A conventional fixed-horizon significance calculation assumes the analysis plan it was designed for; continuing to look until a result appears significant requires a sequential method that accounts for those looks. Otherwise, the nominal threshold does not describe the actual risk of the stopping process.
Independent reader supportYour contribution helps us test, update, and keep practical guides available for everyone.How to design tests that support a decision
- Write the hypothesis and primary metric before launch. Specify the change being tested and the one metric that will determine the main decision. This reduces the temptation to select whichever measure looks best after results arrive.
- Set the decision rule in advance. Choose a significance level and decide whether the test will use a fixed end point or a valid sequential method. Do not change the stopping rule after inspecting results.
- Choose the minimum effect worth acting on. A statistically detectable difference is not automatically valuable. Define the smallest improvement that would justify the cost or risk of implementation, then plan the test around that threshold.
- Estimate sample size, power, and duration before starting. The required traffic and time depend on the baseline, the effect worth detecting, the selected significance level, and desired power. Adobe notes that significance, visitor requirements, and power involve trade-offs; a short test with insufficient sample can miss a real effect.
- Record planned variants and segment comparisons. Decide which comparisons are confirmatory in advance. Treat unplanned segment searches as exploratory, and account for the larger number of comparisons when interpreting them.
- Validate consequential wins before broad deployment. Separate exploration, where ideas are screened, from confirmation, where a decision is tested under a preplanned design. Berman and Van den Bulte discuss two-stage designs with multiple variations as an approach to reducing false discovery; a staged rollout or follow-up test can also limit the cost of acting on a result that does not persist.
These practices make the evidence easier to interpret, but they cannot guarantee that every result is correct. The cost of a false positive versus a false negative should influence the threshold and validation effort: an irreversible, high-impact change warrants more caution than a low-risk experiment that can be rolled back.
Quick Recap
Questions to settle before comparing test options
- How many variants and comparisons are planned? More planned comparisons mean more chances for a nominally significant result to appear by chance; clarify which outcomes are primary and which are exploratory.
- Is the test fixed-horizon or sequential? If results will be reviewed repeatedly, use a method designed for sequential looks rather than applying a fixed-horizon rule at every check.
- Is the test powered for the effect that matters? Plan for the minimum effect that would change the decision, not merely for any detectable difference.
- What is the cost of each kind of error? Balance the cost of deploying an ineffective change against the cost of overlooking a genuine improvement.
Sources and scope
- Ron Berman and Christophe Van den Bulte, “False Discovery in A/B Testing,” Management Science 68(9), 2022. Published online December 30, 2021; the estimates above concern commercial website experimentation data.
- Adobe Experience League, “How Do I Avoid Common A/B Testing Mistakes?”. Official Adobe Target guidance updated September 28, 2026.
- Adobe Experience League, “How Long Should I Run an A/B Test?”. Official guidance on significance level, sample size, statistical power, and test duration.
- Ron Kohavi and Nanyu Chen, “False Positives in A/B Tests,” KDD ’24, 2024. The paper’s search-result description identifies sequential group testing as a proposed way to reduce false positives; no more detailed claim is made here.
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
