P-Values Explained: What 0.05 Does and Does Not Prove
The p-value is probably the most quoted and most misread number in statistics. A result is called "significant" when p falls below 0.05, and that label is often taken to mean the finding is true, important or unlikely to be chance. None of those follows automatically.
This article defines the p-value precisely, works through a coin example you can verify, and lists the misreadings worth avoiding. The aim is to read the number correctly, not to dismiss it.
The definition in plain terms
Start by assuming the null hypothesis: that nothing special is happening, such as a coin being fair or a new design performing the same as the old one. The p-value is then the probability of seeing a result at least as extreme as yours if that assumption were true.
A small p-value means your data would be unusual in a world where the null is true. It is evidence against the null, not a direct statement about how likely the null is.
p = P(data this extreme or more | null hypothesis is true)The condition matters: the probability is calculated inside the null world.
A worked example: 60 heads in 100 flips
You flip a coin 100 times and get 60 heads. Is it biased? Under the null (a fair coin) the expected number of heads is 50, and the standard deviation is √(100 × 0.5 × 0.5) = 5. Your result is 10 heads above expectation, which is z = (60 − 50) ÷ 5 = 2 standard deviations.
For a normal curve, a two-sided tail beyond z = ±2 has probability about 0.0455. By this quick approximation, p is below 0.05.
The exact binomial calculation sums the probabilities of 60 or more heads (about 0.0284) and doubles it for the other tail, giving p ≈ 0.0569, which is above 0.05. The quick normal approximation, used without a continuity correction, gives a slightly smaller p than the exact answer, and here that is enough to change the verdict.
The lesson is not that one method is wrong, but that 0.0455 and 0.0569 describe nearly the same evidence. A hard cut-off at 0.05 turns a tiny difference into opposite conclusions. You can compute either version with the p-value calculator or the binomial distribution calculator.
One-sided or two-sided?
A two-sided test counts extreme results in both directions, here too many heads or too many tails. A one-sided test counts only one direction. For z = 2 the one-sided tail is 0.0228, exactly half of the two-sided 0.0455.
The choice must be made before seeing the data, and the one-sided version is justified only when a result in the other direction would genuinely be treated the same as no effect. Switching to one-sided after the fact simply halves p, which is a quiet way of manufacturing significance.
Five common misreadings
- "p = 0.03 means a 3% chance the result is due to chance." No. It is the chance of data this extreme if the null is true, not the chance that the null is true.
- "p > 0.05 means there is no effect." No. It means the data did not give strong evidence against the null. A small study can miss a real effect.
- "A smaller p means a bigger effect." No. A huge sample can give a tiny p for an effect too small to matter in practice.
- "p < 0.05 means the finding will replicate." No. It is one piece of evidence from one dataset.
- "0.05 is a law of nature." No. It is a convention. Fields choose stricter or looser thresholds depending on the cost of being wrong.
The multiple-testing trap
A 0.05 threshold accepts a 5% false-alarm rate on each test when the null is true. Run many tests and those chances accumulate. For independent tests, the probability of at least one false positive is 1 − 0.95ⁿ.
| Number of tests | Chance of at least one false positive |
|---|---|
| 1 | 5.0% |
| 5 | 22.6% |
| 10 | 40.1% |
| 20 | 64.2% |
That is why looking at twenty subgroups or twenty metrics and reporting the one that "worked" is unreliable. Decide on the tests in advance, or use a correction that tightens the threshold as the number of tests grows.
What to report alongside the p-value
- The effect size, such as a difference in means or a percentage-point change, so readers can judge whether it matters.
- A confidence interval, which shows a plausible range for the effect rather than a yes/no verdict.
- The sample size and how it was collected.
- The exact p-value (for example 0.057) instead of only "not significant".
If you already have a test statistic and need the matching probability, the z-score to p-value calculator converts one into the other.
- P-value Calculator: Compare two groups, such as a control and a variant, and find out how likely a difference this large would be if there were no real effect. Enter proportions or means, pick a Z-test or a T-test, and read off the p-value and significance level.
- Z-score to P-value Calculator: Enter a Z-score, choose a one-tailed or two-tailed test and get the matching p-value from the standard normal distribution. The guide explains how to read the number.
- Binomial Distribution Calculator: Enter the chance of success on a single trial, how many trials are run and how many successes you are asking about. The result is the exact probability together with four cumulative ones, and a second tab works backwards to find the trials required.
This article is general information, not professional advice. See our Disclaimer and Editorial Policy.
More articles
How to Add and Subtract Fractions With Different Denominators
You cannot add 1/2 and 1/3 by adding the tops and bottoms. This guide shows the common-denominator method, with worked examples and mixed numbers.
Common Probability Mistakes and How to Avoid Them
Probability is full of results that feel wrong. Here are five classic mistakes, each with a short calculation showing the correct answer.
Converting Between Fractions, Decimals and Percentages
Fractions, decimals and percentages are three ways of writing the same quantity. Learn the six conversions, repeating decimals and the equivalents worth memorising.