I'm analyzing a dataset with over a million rows. My t-tests are returning p-values like 0.0001, but the actual difference between the groups is negligible. Is this just a result of the large sample size, and should I be focusing more on Cohen's d or other effect size metrics instead of just significance?
3 answers
This is a classic phenomenon in statistical analysis often called the "Large Sample Size Trap." With a million rows, even the tiniest, most trivial difference becomes statistically significant because the standard error becomes incredibly small. The p-value only tells you that the difference is likely not due to chance, not that the difference is important. You absolutely must report Cohen's d or R-squared to show the practical significance. If your Cohen's d is below 0.2, it's a "small" effect, regardless of how many zeros are in your p-value. Focus on the business impact.
Have you tried performing a power analysis or perhaps sub-sampling your data to see if the significance holds up with a smaller, more manageable group?
The p-value tells you about the noise, but the effect size tells you about the signal. Always lead with the effect size when talking to executives or stakeholders.
Well said, Steven. I've seen many projects fail because they chased a "significant" p-value that had zero actual impact on the company's bottom line or user experience.
Thomas, I did try sub-sampling with 5,000 records and the p-value jumped to 0.15, which confirmed my suspicion. Is it common practice to sub-sample for the sake of making p-values more "realistic," or is that considered bad form in professional data science?