Understanding Confidence Intervals and P-Values

Few statistical concepts are as widely reported, and as widely misinterpreted, as p-values and confidence intervals. Both describe uncertainty in a result — but not in the way most people first assume.

What a p-value actually is

The p-value is the probability of observing a result at least as extreme as yours, if the null hypothesis were true. It is not the probability that the null hypothesis is true, and it is not the probability that your finding happened by chance. A p-value of 0.03 doesn't mean there's a 3% chance the result is a fluke — it means that if there really were no effect, you'd see data this extreme (or more) 3% of the time.

What a confidence interval actually is

A 95% confidence interval means: if you repeated the study many times, 95% of the intervals calculated this way would contain the true population value. It does not mean there's a 95% probability the true value falls within this specific interval — a subtle but important distinction.

Why confidence intervals often tell you more

A p-value gives you a single yes/no signal. A confidence interval shows you the range of plausible effect sizes and their precision — a narrow interval far from zero is more informative than "p < 0.05" alone, and a wide interval crossing zero (or 1, for ratio measures) tells you the result is imprecise even if technically "significant."

Common reporting mistakes

• Treating p = 0.049 as meaningfully different from p = 0.051 (it isn't).
• Reporting only p-values without effect sizes or confidence intervals, leaving readers unable to judge practical significance.
• Interpreting "not statistically significant" as "no effect exists," when it may just mean the study was underpowered.