The two ways a test can be wrong
A hypothesis test starts from a null hypothesis, H0 — the claim of no effect, no difference, no relationship — and you decide from data whether to reject it. Because the null is either true or false in reality, and you either reject it or fail to reject it, there are four possible outcomes: two correct ones and two mistakes.
A Type I error is rejecting a null hypothesis that is actually true. You announce an effect that is not there — a false positive. A Type II error is failing to reject a null hypothesis that is actually false. A real effect exists and your test missed it — a false negative. Medical screening is the cleanest analogy: a Type I error tells a healthy patient they are sick, a Type II error tells a sick patient they are fine. Note the asymmetry in the language too — you never 'accept' the null, you only fail to reject it, because absence of evidence is not evidence of absence.
- Type I error: rejecting H0 when H0 is true — a false positive, claiming an effect that is not real.
- Type II error: failing to reject H0 when H0 is false — a false negative, missing a real effect.
- The two correct outcomes: correctly rejecting a false H0, and correctly failing to reject a true H0.
Alpha: the false positive rate you choose
Alpha (α) is the probability of a Type I error, and it is the one number here you set yourself. Picking α = 0.05 before collecting data says: if the null hypothesis is true, I am willing to reject it wrongly about 5% of the time. Alpha is also the significance level you compare the p-value against, which is why the two ideas keep appearing together.
Because you choose it, alpha states your tolerance for false alarms rather than a property of your data. Fields that pay a heavy price for false positives tighten it to 0.01 or lower; exploratory screening sometimes loosens it. One consequence catches people out: alpha applies to each test you run. Test twenty unrelated hypotheses at α = 0.05 and you should expect roughly one false positive by chance alone, even if nothing real is going on. That is why multiple comparisons need an adjustment rather than an apology.
- α = P(reject H0 | H0 is true) — the long-run false positive rate.
- You choose alpha in advance; it is a tolerance, not a measurement.
- Alpha applies per test, so running many tests multiplies the expected number of false positives.
Beta and statistical power
Beta (β) is the probability of a Type II error: the chance your test fails to detect a real effect. Unlike alpha, you do not set beta directly. It falls out of four things — how big the true effect is, how variable your data is, how many observations you collected, and what alpha you chose. It is therefore never a single number for a study; it is always quoted against a specific effect size worth detecting.
Power is the more useful way to say the same thing. Power = 1 − β, the probability that your test correctly rejects a false null — that it finds the effect when the effect is real. A study with 80% power against a given effect size will detect it about four times in five and miss it once. That 80% convention is a working floor, not a rule of nature; 90% is preferable when a missed effect is costly.
Low power is the quiet failure mode of applied statistics. An underpowered study reporting 'no significant difference' has not shown the effect is absent — it may simply have been unable to see it, which is why a non-significant result should always be reported with a confidence interval.
- β = P(fail to reject H0 | H0 is false); power = 1 − β.
- Power is always stated relative to a specific effect size worth detecting.
- A non-significant result from a low-power study is uninformative, not evidence of no effect.
The trade-off, and what actually buys you both
For a fixed study design, alpha and beta pull against each other. Lowering alpha from 0.05 to 0.01 raises the evidence bar, so you reject the null less often — cutting false positives but raising false negatives and lowering power. Raising alpha does the reverse. Moving the threshold only slides errors between the two piles; it never shrinks both at once. What does shrink both is a better design: more data, less measurement noise, or a larger true effect all push the sampling distributions under the null and the alternative further apart, buying power without spending any alpha.
Here is that in numbers, using a made-up illustration. Test H0: μ = 100 against the one-sided alternative μ > 100 at α = 0.05, with population standard deviation 15 and a true mean of 105. At n = 36 the standard error is 2.5, the cutoff sits at 104.11, and power is about 0.64 — so β ≈ 0.36 and you miss this real effect more than a third of the time. Raise n to 100 and the standard error falls to 1.5, the cutoff drops to 102.47, and power rises to about 0.95 with β ≈ 0.05. Alpha never moved; only the sample size did. That is why power analysis belongs before data collection, not after.
- At a fixed design, lowering alpha raises beta and lowers power, and vice versa.
- Larger n, larger true effect, and lower variability all raise power without changing alpha.
- In the worked illustration above, going from n = 36 to n = 100 lifted power from about 0.64 to about 0.95.
Choosing the balance, and keeping the two straight
The right balance is a judgement about consequences, not a statistical question. Ask which mistake costs more. If a false positive triggers an expensive rollout or an unnecessary treatment, tighten alpha. If a false negative means a real safety signal goes unnoticed, protect power instead — by planning a larger sample, not by loosening the threshold.
Two habits keep the ideas straight. Alpha is conditional on the null being true and beta on the null being false, so they are not two slices of one pie and never sum to 1. And both are long-run rates for a procedure, not the probability that this conclusion is wrong. For the ordering, a mnemonic helps: Type I is being too eager, crying wolf when there is no wolf; Type II is being too cautious, missing the wolf that is really there.
To watch the trade-off move rather than memorize it, open StatRise's Type I & II Errors simulation and drag the alpha and effect-size controls. Both that simulation and the step-by-step hypothesis-test calculators sit behind Premium, which unlocks all 39 calculators, every lesson, all 660 practice questions and all six simulations; the formula reference, glossary and topic pages stay free.
- Pick alpha from the cost of a false alarm; buy power with sample size, not by relaxing alpha.
- Alpha and beta are conditional on opposite states of the world, so they never sum to 1.
- Type I = crying wolf; Type II = missing the wolf.