Mistake Master
You can fail to reject. You can never accept
The arithmetic of a test is short; the sentence at the end is where the marks are. Two asymmetries govern it. Evidence can count against a claim and never for it, so a large p-value leaves the null standing without supporting it. And a small p-value says an effect is detectable, not that it is large: with enough data, a difference too small to act on will clear any threshold you like.
§1
Four steps, and the third is the shortest.
▸
A complete test has four parts:
- State: define the parameter in context, write $H_0$ and $H_a$, and give $\alpha$.
- Plan: name the procedure (one-proportion z-test) and check the conditions with the study's numbers.
- Do: compute the test statistic and the p-value.
- Conclude: compare the p-value to $\alpha$, make the decision, and state what it means about the population in context.
Continuing the renewal example: $H_0: p = 0.40$ against $H_a: p < 0.40$, $\alpha = 0.05$, $n = 250$ with 88 renewals. Conditions: random sample stated, $250 \le 0.10N$, and $np_0 = 100$, $n(1-p_0) = 150$. Then
$$z = \frac{0.352 - 0.40}{\sqrt{\frac{0.40(0.60)}{250}}} \approx -1.55, \qquad \text{p-value} \approx 0.061.$$
Since $0.061 > 0.05$, we fail to reject $H_0$. There is not convincing evidence that the renewal rate among all customers is below 40%.
§2
Two decisions exist, and one of them is not a verdict for the null.
▸
The rule is mechanical: reject $H_0$ when the p-value is less than $\alpha$, and fail to reject $H_0$ otherwise. There is no third option, and in particular there is no "accept $H_0$".
The asymmetry is real, not a wording convention. A small p-value says the data would be unusual if the null were true, which is a reason to doubt the null. A large p-value says the data are the kind the null routinely produces, and so would many nearby values of $p$: the renewal data above are also consistent with $p = 0.38$, $p = 0.36$, and a range of other values. Failing to rule something out is not evidence that it is correct.
Three sentences to strike: "we accept $H_0$", "the data prove there is no difference", and "the renewal rate is 40%". Each converts an absence of evidence into a positive finding. The correct form is: there is not convincing evidence that $p$ is below 0.40.
The same discipline applies on the other side. Rejecting $H_0$ is not proof that $H_a$ is true; it means the data would be unusual under the null, at the stated level.
§3
Significant means detectable, not large.
▸
Statistical significance answers whether an effect can be distinguished from chance. It says nothing about size, and the two come apart in both directions once sample size enters.
- Tiny effect, enormous sample. With $n = 40{,}000$ and $\hat{p} = 0.508$ against $p_0 = 0.50$, the standard error is 0.0025, so $z = 3.2$ and the two-sided p-value is about 0.0014. Highly significant, and the effect is eight tenths of a percentage point.
- Large effect, small sample. With $n = 30$ and $\hat{p} \approx 0.633$ against $p_0 = 0.50$, the standard error is 0.091, so $z \approx 1.46$ and the one-sided p-value is about 0.072. Not significant at 0.05, and the observed gap is more than 13 percentage points.
So a significant result deserves a follow-up question about magnitude, usually answered with a confidence interval, and a non-significant result deserves a question about whether the sample was large enough to detect an effect worth detecting.
The threshold is not a cliff either. A p-value of 0.049 and one of 0.051 describe nearly identical evidence, and the decision differs only because a line was drawn at 0.05. Reporting the p-value itself, rather than only the verdict, is what keeps that visible.
§4
The conclusion names the population and keeps its hedge.
▸
A conclusion has three components, and a missing one costs marks:
- Compare: "Since the p-value of 0.061 is greater than $\alpha = 0.05$, ..."
- Decide: "... we fail to reject $H_0$."
- Interpret in context: "There is not convincing evidence that the proportion of all the company's customers who renew is less than 0.40."
Two habits carry it. Name the population, since a conclusion about the 250 surveyed customers is about a group whose renewal rate is already known exactly. And keep the language hedged: convincing evidence, evidence that, never proves, shows, or confirms. The hedge is what makes the sentence a true report of a sample-based decision that can be wrong, which is exactly the subject of the next topic.
§5
Skill Check.
▸
Ten scenarios. Pick the chips that match your answer, then check. A scenario marks complete the first time every part is right. Progress saves on this device.