Mistake Master
Student view — seeing the site as a student does
Home Unit 3 · Inference for Categorical Data: Proportions 3.1·3.2·3.3·3.4·3.5·3.6·3.7·3.8·3.9·3.10·3.11·3.12·3.13·3.14·3.15 Lesson
Skill Check 0 / 10 complete

You can fail to reject. You can never accept

The arithmetic of a test is short; the sentence at the end is where the marks are. Two asymmetries govern it. Evidence can count against a claim and never for it, so a large p-value leaves the null standing without supporting it. And a small p-value says an effect is detectable, not that it is large: with enough data, a difference too small to act on will clear any threshold you like.

§1

Four steps, and the third is the shortest.

A complete test has four parts:

  1. State: define the parameter in context, write $H_0$ and $H_a$, and give $\alpha$.
  2. Plan: name the procedure (one-proportion z-test) and check the conditions with the study's numbers.
  3. Do: compute the test statistic and the p-value.
  4. Conclude: compare the p-value to $\alpha$, make the decision, and state what it means about the population in context.

Continuing the renewal example: $H_0: p = 0.40$ against $H_a: p < 0.40$, $\alpha = 0.05$, $n = 250$ with 88 renewals. Conditions: random sample stated, $250 \le 0.10N$, and $np_0 = 100$, $n(1-p_0) = 150$. Then

$$z = \frac{0.352 - 0.40}{\sqrt{\frac{0.40(0.60)}{250}}} \approx -1.55, \qquad \text{p-value} \approx 0.061.$$

Since $0.061 > 0.05$, we fail to reject $H_0$. There is not convincing evidence that the renewal rate among all customers is below 40%.

§2

Two decisions exist, and one of them is not a verdict for the null.

The rule is mechanical: reject $H_0$ when the p-value is less than $\alpha$, and fail to reject $H_0$ otherwise. There is no third option, and in particular there is no "accept $H_0$".

The asymmetry is real, not a wording convention. A small p-value says the data would be unusual if the null were true, which is a reason to doubt the null. A large p-value says the data are the kind the null routinely produces, and so would many nearby values of $p$: the renewal data above are also consistent with $p = 0.38$, $p = 0.36$, and a range of other values. Failing to rule something out is not evidence that it is correct.

Three sentences to strike: "we accept $H_0$", "the data prove there is no difference", and "the renewal rate is 40%". Each converts an absence of evidence into a positive finding. The correct form is: there is not convincing evidence that $p$ is below 0.40.

The same discipline applies on the other side. Rejecting $H_0$ is not proof that $H_a$ is true; it means the data would be unusual under the null, at the stated level.

§3

Significant means detectable, not large.

Statistical significance answers whether an effect can be distinguished from chance. It says nothing about size, and the two come apart in both directions once sample size enters.

  1. Tiny effect, enormous sample. With $n = 40{,}000$ and $\hat{p} = 0.508$ against $p_0 = 0.50$, the standard error is 0.0025, so $z = 3.2$ and the two-sided p-value is about 0.0014. Highly significant, and the effect is eight tenths of a percentage point.
  2. Large effect, small sample. With $n = 30$ and $\hat{p} \approx 0.633$ against $p_0 = 0.50$, the standard error is 0.091, so $z \approx 1.46$ and the one-sided p-value is about 0.072. Not significant at 0.05, and the observed gap is more than 13 percentage points.

So a significant result deserves a follow-up question about magnitude, usually answered with a confidence interval, and a non-significant result deserves a question about whether the sample was large enough to detect an effect worth detecting.

The threshold is not a cliff either. A p-value of 0.049 and one of 0.051 describe nearly identical evidence, and the decision differs only because a line was drawn at 0.05. Reporting the p-value itself, rather than only the verdict, is what keeps that visible.

§4

The conclusion names the population and keeps its hedge.

A conclusion has three components, and a missing one costs marks:

  1. Compare: "Since the p-value of 0.061 is greater than $\alpha = 0.05$, ..."
  2. Decide: "... we fail to reject $H_0$."
  3. Interpret in context: "There is not convincing evidence that the proportion of all the company's customers who renew is less than 0.40."

Two habits carry it. Name the population, since a conclusion about the 250 surveyed customers is about a group whose renewal rate is already known exactly. And keep the language hedged: convincing evidence, evidence that, never proves, shows, or confirms. The hedge is what makes the sentence a true report of a sample-based decision that can be wrong, which is exactly the subject of the next topic.

§5

Skill Check.

Ten scenarios. Pick the chips that match your answer, then check. A scenario marks complete the first time every part is right. Progress saves on this device.

0 of 10 scenarios complete