Mistake Master
Student view — seeing the site as a student does
Home› Unit 3 · Inference for Categorical Data: Proportions› 3.1·3.2·3.3·3.4·3.5·3.6·3.7·3.8·3.9·3.10·3.11·3.12·3.13·3.14·3.15

Potential Errors When Performing Tests

▶︎  Watch it animatedinteractive step-through · ~3 min · optional

Crossing the truth with the decision gives two errors: a Type I error rejects a true null (a false alarm) and a Type II error fails to reject a false one (a missed detection). Their probabilities are conditional on different states, $\alpha = P(\text{reject} \mid H_0 \text{ true})$ and $\beta = P(\text{fail to reject} \mid H_0 \text{ false})$, so the significance level is the Type I error rate, while $\beta$ and power $= 1 - \beta$ are quoted against a specific alternative value. Moving the cutoff trades one for the other; only a larger sample lowers both, by narrowing the two distributions so they overlap less.

The errors get swapped and their consequences land on the wrong cell: a missed real effect called a Type I error, $\alpha$ described as the Type II rate, or an error named without reference to the null at all. Lowering $\alpha$ gets described as making the test more accurate, when it only shifts mistakes from false alarms to missed detections. And power gets treated as a property of the test alone rather than a statement about a particular alternative, which is what makes a non-significant result from a small sample uninformative rather than reassuring.

truth (rows) against decision (columns) fail to reject H0 reject H0 H0 is TRUE correct TYPE I error false alarm, P = alpha H0 is FALSE TYPE II error missed, P = beta correct P = power = 1 - beta
Both errors are defined against the null, so naming one requires stating the null first. Only one row is real in any given situation, so only one of the two errors is available to be made.
left: if H0 is true right: if the alternative value is true cutoff alpha beta power slide the cutoff right: alpha falls, beta rises. left: the reverse only a larger n narrows BOTH curves, so both errors fall together
The cutoff cannot shrink both shaded regions at once, which is what makes the choice of alpha a judgment about which mistake costs more. Sample size is the only lever that moves both.

The work

3 ways in · any order
Lesson
Potential Errors When Performing Tests ›

Lays out the two-by-two table of truth against decision, defines alpha and beta as conditional probabilities on different states, shows why the cutoff trades one error for the other, and lists the four things that raise power along with the one that is not free.

Skill check · 10 scenarios
Open lesson →
Diagnostic
10-item topic check ›

Ten items on test errors: the two types swapped, consequences attached to the wrong cell, alpha called the Type II rate, and lowering alpha described as reducing both errors. Take it cold to find your habit, or after the lesson to check it is gone.

Not started · 10 items · ~15 min
Start diagnostic →
Targeted Practice
Drill a single misconception ›

Pick one of the failure modes you missed and drill it on its own. The round is adaptive: two correct in a row clears it for now and moves you to the next. Two in a row is a checkpoint, not proof: if the error resurfaces later, the misconception comes back.

Take the diagnostic to identify your misconceptions
Diagnostic first