The correct definition
A p-value is the probability of observing data at least as extreme as what you actually got, assuming the null hypothesis is true. Read that carefully: it is a probability about the data, computed under the assumption that there is no real effect. It is not the probability that the null hypothesis is true, and it is not the probability that your result happened by chance in some general sense.
A concrete way to feel it: suppose a coin is fair (the null hypothesis) and you flip it 20 times and get 18 heads. The p-value asks, 'if this coin really were fair, how often would I see a result this lopsided or more?' That probability is tiny, so a fair coin would rarely produce your data — which is evidence the coin is not fair. The smaller the p-value, the more your data conflicts with the null.
- p-value = P(data this extreme or more | null hypothesis is true).
- It is a statement about the data under the null, not about the hypothesis itself.
- Smaller p-value = data is more surprising under the null = stronger evidence against it.
How the p-value drives the decision
In a hypothesis test, you compare the p-value to your pre-chosen significance level, alpha, which is usually 0.05. If the p-value is less than or equal to alpha, you reject the null hypothesis and call the result statistically significant. If it is greater than alpha, you fail to reject the null. That is the entire decision rule.
The threshold is a convention, not a law of nature. A p-value of 0.049 and one of 0.051 are almost identical in evidential terms, yet they fall on opposite sides of the 0.05 line. This is why many statisticians prefer to report the exact p-value and treat it as a continuous measure of evidence rather than a hard pass/fail gate. Think of it as a dial, not a switch.
- Decision rule: p ≤ α reject the null; p > α fail to reject.
- 0.05 is a convention — 0.049 and 0.051 mean almost the same thing.
- Report the exact p-value; treat it as a continuous strength-of-evidence measure.
What a p-value is NOT
Most p-value errors are the same handful of misreadings, so it helps to name them directly. A p-value is not the probability that the null hypothesis is true. It is not the probability that your results are due to chance. A p-value of 0.05 does not mean there is a 95% chance the alternative hypothesis is correct. And a small p-value says nothing about how large or important the effect is — only about how incompatible your data is with the null.
The size-versus-significance confusion is especially costly. With a large enough sample, a trivially small, meaningless effect can produce a very small p-value, because large samples can detect tiny departures from the null. Statistical significance answers 'is there likely some effect at all', while effect size and confidence intervals answer 'how big is it, and does it matter'. You need both.
- Not the probability the null (or alternative) is true.
- Not the probability your result is 'due to chance'.
- Not a measure of effect size — a tiny effect can be significant with a big sample.
One-tailed vs two-tailed p-values
How you compute the p-value depends on your alternative hypothesis. A two-tailed test, where the alternative is 'not equal', counts extreme results in both directions and splits the probability across both tails. A one-tailed test, where the alternative specifies a direction ('greater than' or 'less than'), counts extremes in only one tail, which makes it easier to reach significance in that direction.
Because a one-tailed test is more permissive, you must justify the direction from the science before seeing the data, never switch to one-tailed just to squeeze under 0.05. When in doubt, the two-tailed test is the more conservative and defensible choice. The calculators in StatRise let you specify the tail so the p-value matches the hypothesis you actually set up.
- Two-tailed: alternative is 'not equal'; probability is split across both tails.
- One-tailed: alternative is directional; all probability is in one tail.
- Choose the tail from the hypothesis in advance — never to chase significance.
Using p-values responsibly
A p-value is one piece of evidence, not a verdict. Good statistical practice pairs it with an effect size and a confidence interval so you know both whether an effect is detectable and how large it plausibly is. It also treats a single significant result with humility: p-values vary from sample to sample, and results that do not replicate are common when studies are small or when many comparisons are run without adjustment.
Beware of 'p-hacking' — running many tests and reporting only the ones that cross 0.05. If you test twenty unrelated things at α = 0.05, you expect about one false positive just by chance. The remedy is to plan your analysis in advance, report everything you tried, and interpret borderline results cautiously. To build real intuition, compute p-values for many worked problems: the StatRise practice bank and step-by-step calculators are a good place to see how the number behaves across different tests.
- Report a p-value alongside an effect size and confidence interval.
- Expect variation between samples; one significant result is not proof.
- Avoid p-hacking — running many tests inflates false positives.