Confusing correlation with causation
The most famous statistical error is also the most persistent: assuming that because two variables move together, one causes the other. Ice cream sales and drowning deaths rise together, but ice cream does not cause drowning — hot weather drives both. A correlation can arise from a genuine causal link, a reversed causal link, a shared third cause (a confounder), or pure coincidence, and the correlation alone cannot tell you which.
Establishing causation generally requires a controlled experiment with randomization, which breaks the influence of confounders, or careful methods designed to rule alternatives out. When you only have observational data, describe the relationship as an association and resist causal language. Ask yourself what third variable might explain both before drawing any conclusion.
- Correlation can come from causation, reverse causation, a confounder, or coincidence.
- Randomized experiments are the standard route to causal claims.
- With observational data, say 'associated with', not 'causes'.
Misreading the p-value
P-value errors are so common they deserve their own list. A p-value is not the probability that the null hypothesis is true, not the probability your result is due to chance, and not a measure of how big an effect is. It is the probability of data at least as extreme as yours if the null were true — nothing more. Treating a small p-value as proof of a large or important effect is a frequent and costly slip.
A related error is treating 0.05 as a magic line, so that 0.049 is celebrated and 0.051 dismissed, when the two carry nearly identical evidence. Report exact p-values, pair them with effect sizes and confidence intervals, and remember that a single significant result is evidence, not certainty.
- The p-value is P(data this extreme | null true) — not the probability the null is true.
- Significance is not effect size; a tiny effect can be significant with a big sample.
- Do not treat 0.05 as a hard cliff — report the exact value.
Ignoring the test's assumptions
Every test rests on assumptions, and a valid test applied to data that violates them can produce a confidently wrong answer. Common assumptions include independence of observations, approximate normality (or a large enough sample), and, for some tests, roughly equal variances between groups. Chi-square tests additionally assume expected cell counts are not too small.
The fix is to check before you trust. Look at whether observations are truly independent, whether the sample is large enough for the Central Limit Theorem to help, and whether a histogram of your data is wildly skewed. When assumptions fail badly, a nonparametric alternative or a transformation may be more appropriate than forcing the standard test.
- Check independence, normality or sample size, and equal-variance assumptions.
- For chi-square, verify expected counts are large enough.
- When assumptions fail, consider nonparametric tests or transformations.
Sampling and selection problems
A statistic is only as good as the sample behind it. A biased sample — one that systematically over- or under-represents part of the population — produces estimates that no amount of clever analysis can rescue. Convenience samples, voluntary-response surveys (where only people with strong opinions reply), and survivorship bias (studying only the survivors) all quietly distort conclusions.
Small samples cause a different problem: high variability and low power, so real effects go undetected and flukes look impressive. Before believing a result, ask how the data was collected and whether the sample plausibly represents the population you care about. A representative sample of modest size usually beats a large but biased one.
- Biased samples cannot be fixed by analysis — how data is collected matters most.
- Watch for convenience, voluntary-response, and survivorship bias.
- Small samples have low power; treat lone results from tiny studies cautiously.
Multiple comparisons and cherry-picking
If you run one test at α = 0.05, you accept a 5% false-positive rate. If you run twenty tests, you expect about one false positive by chance alone — so reporting only the 'significant' result from a large fishing expedition is deeply misleading. This is the multiple-comparisons problem, and it underlies a lot of results that fail to replicate.
The defenses are to plan your primary analysis in advance, report every test you ran rather than only the winners, and apply a correction (such as adjusting the threshold) when you genuinely must run many tests. More broadly, treat statistics as a tool for honest reasoning, not for manufacturing significance. Working through many varied problems — as in the StatRise practice bank — builds the judgment to recognize these traps in your own work.
- Running many tests inflates false positives — about 1 in 20 at α = 0.05.
- Report all tests you ran, not just the significant ones.
- Pre-plan the main analysis and correct for multiple comparisons when needed.