We just finished an A/B test on our landing page, and the software gave us a p-value of 0.03. My manager wants to know if this "proves" the new design is better. Is a 3% chance of error low enough to justify a full site redesign? I want to make sure I am not misinterpreting statistical significance when the stakes for our conversion rate are this high.
3 answers
A p-value of 0.03 means that if there were actually no difference between the designs, there’s only a 3% chance you’d see results this extreme. Since it's below the standard 0.05 threshold, it is "statistically significant." However, "significant" doesn't always mean "important." You also need to look at the "Effect Size." If the new design only increased conversions by 0.1%, a redesign might not be worth the cost, even if the result is statistically valid. Always weigh the p-value against the practical business impact and the cost of implementation.
What was your sample size for this test? I've seen p-values look great on small samples only to vanish when we scaled up. Did you run a power analysis before starting to see how much data you actually needed?
It suggests the result isn't just luck. But before you redesign everything, check if the "winner" actually resulted in more sales, not just more clicks.
Exactly, Susan. Statistical significance is just the first hurdle. Business relevance—like actual ROI—is the finish line. Don't let a low p-value blind you to the bottom line.
Mark brings up a vital point, Jennifer. If the sample size was too small, you might be looking at a "false positive." A power analysis ensures that your test is sensitive enough to detect a real difference. If you had 10,000 visitors per variant, that 0.03 is much more trustworthy than if you only had 100. Always check your confidence intervals to see the range of the expected improvement.