Mistake Master
Every p-value carries an assuming
A p-value answers one question: if the null hypothesis were true, how often would a sample look at least this extreme. The clause at the front of that sentence is the whole idea, and every standard misreading is what remains after it gets dropped. A p-value is not the probability the null is true, and no calculation in this course ever produces that number.
§1
The definition, with the clause that makes it mean something.
▸
The p-value is the probability, computed assuming the null hypothesis is true, of obtaining a statistic at least as extreme as the one observed, in the direction the alternative specifies.
Three parts, each load-bearing:
- Assuming the null is true. The entire calculation happens inside a world where $p = p_0$. That world supplies the sampling distribution the probability is read from.
- At least as extreme. Not the probability of exactly this sample, which for a continuous model is 0, but of this result or anything further out.
- In the direction the alternative specifies. The alternative decides which tail, or both.
A small p-value means the observed data would be unusual if the null were true, which is evidence against the null. A large p-value means the data are the kind of thing the null routinely produces, which is not evidence for the null: it is an absence of evidence against it. That distinction is the subject of Topic 3.7 and it starts here.
§2
It is a tail area of the null distribution.
▸
Mechanically, a p-value is an area. Build the sampling distribution of $\hat{p}$ under the null, mark the observed value, and shade everything at least that far out.
A company claims $p = 0.40$ and a manager suspects less, so $H_0: p = 0.40$ against $H_a: p < 0.40$. Of 250 customers, 88 renewed, giving $\hat{p} = 0.352$. Under the null,
$$\sigma = \sqrt{\frac{0.40(0.60)}{250}} \approx 0.0310, \qquad z = \frac{0.352 - 0.40}{0.0310} \approx -1.55.$$
The p-value is the area to the left of $z = -1.55$, about $0.061$. Read back in words: if the renewal rate really were 40%, about 6% of random samples of 250 customers would produce a renewal proportion of 0.352 or lower. The observed sample is on the low side of what the null predicts, but not by an amount the null would rarely produce.
Everything about the arithmetic depends on the null being assumed. That is where the 0.40 in the standard error comes from, and it is why the distribution is centered at 0.40 rather than at the observed 0.352.
§3
The alternative decides which area counts.
▸
What counts as "at least as extreme" changes with the alternative:
- $H_a: p < p_0$: the area to the left of the observed statistic.
- $H_a: p > p_0$: the area to the right.
- $H_a: p \ne p_0$: both tails, since a result equally far out on the other side would be equally surprising. For a symmetric null distribution, that doubles the one-sided value.
In the renewal example the two-sided p-value would be $2(0.061) \approx 0.121$. Same data, same test statistic, different question, and the two-sided version is harder to call significant precisely because it counts surprise in both directions.
Reporting a one-sided p-value for a two-sided question halves the number and doubles the apparent evidence, which is why the alternative has to be fixed before the data are seen. And a p-value can never exceed 1 or fall below 0: it is a probability, so an answer outside that range means a tail was computed on the wrong side or doubled when it should not have been.
§4
Four sentences that drop the condition.
▸
Each of these is a p-value of 0.061 misread, and each drops the assuming-the-null clause:
- "There is a 6.1% chance the null is true." The p-value is computed given the null, so it cannot also be a probability about the null. That would be the reverse conditional, and Topic 2.6 showed those are different numbers.
- "There is a 93.9% chance the alternative is true." The complement of a misreading is still a misreading. Nothing here assigns a probability to a hypothesis.
- "There is a 6.1% chance the results are due to chance." The whole calculation already assumes chance alone is at work; that is what the null world is. The p-value says how often chance alone produces a result this extreme.
- "The probability of getting exactly this sample is 0.061." The definition says at least as extreme, and the exact-value probability is a different and much smaller quantity.
The repair in every case is to restore the clause and name the direction: "if $p$ really were 0.40, about 6% of samples of this size would give a renewal proportion of 0.352 or lower." A sentence with no if in it is not yet an interpretation of a p-value.
§5
Skill Check.
▸
Ten scenarios. Pick the chips that match your answer, then check. A scenario marks complete the first time every part is right. Progress saves on this device.