Mistake Master
Zero inside means undecided
For a difference, one value carries all the weight: zero, the claim that the two population proportions are equal. Where the interval sits relative to zero decides what can be said, and the sign of the endpoints decides which group is ahead, but only once the order of subtraction has been written down. An interval containing zero has not shown the groups are the same; it has failed to show they differ, which is a different sentence.
§1
Zero is the no-difference value.
▸
An interval for $p_1 - p_2$ is judged against 0, because $p_1 - p_2 = 0$ says the two population proportions are equal. Three positions are possible:
- Entirely above 0: every plausible difference is positive, so there is evidence that $p_1 > p_2$.
- Entirely below 0: every plausible difference is negative, so there is evidence that $p_1 < p_2$.
- Containing 0: zero is plausible, and so are differences in both directions, so the data do not establish which proportion is larger.
The checkout interval $(0.018, 0.182)$ for new minus old lies entirely above 0, so there is convincing evidence that the new system's completion rate is higher, by somewhere between about 2 and 18 percentage points.
§2
Containing zero is a failure to decide, not a finding of equality.
▸
An interval of $(-0.02, 0.09)$ contains 0. What it establishes is that the data are consistent with no difference, and equally consistent with $p_1$ exceeding $p_2$ by up to about 9 percentage points, and with $p_2$ exceeding $p_1$ by up to about 2.
Two conclusions are therefore unavailable:
- "The two proportions are equal" or "the groups are the same". Zero is one plausible value among many, with no special standing beyond being the value the comparison happens to be judged against.
- "Group 1 is larger, since most of the interval is positive". How much of the interval sits on each side is not the criterion. Values below 0 are inside, so a negative difference remains plausible.
The correct sentence names the failure: these data do not provide convincing evidence of a difference between the two population proportions. That is a real result, and it is the same asymmetry as failing to reject a null: absence of evidence is not evidence of absence.
§3
The sign means nothing without the order of subtraction.
▸
An interval of $(-0.14, -0.02)$ says the first group's proportion is lower than the second's, by between 2 and 14 percentage points. Which group that is depends entirely on which was labeled first, and a report that omits the order is unreadable.
Two habits prevent the confusion. State the order in words at the top, "treatment minus control", and repeat it in the conclusion. And check the sign against the point estimate: if $\hat{p}_1 = 0.30$ and $\hat{p}_2 = 0.44$, the difference is $-0.14$ and the interval must be centered near there, so an interval sitting on the positive side is a signal that the subtraction got reversed somewhere.
Reversing the order flips both endpoints and their signs, turning $(-0.14, -0.02)$ into $(0.02, 0.14)$. The two intervals carry identical information, and a reader can only tell them apart from the stated order.
§4
The justification, with the extra sentence a difference needs.
▸
Four parts, and the third is the one unique to differences:
- State: "We are 95% confident the interval from 0.018 to 0.182 captures the true difference, new minus old, in the proportion of all shoppers completing a purchase."
- Locate 0: "The interval lies entirely above 0."
- Give the direction: "So the new system's rate is higher."
- Give the magnitude in context: "by roughly 2 to 18 percentage points."
Keep the hedge throughout: convincing evidence, plausible, never proves. And keep the causal claim out unless the design supports it. If shoppers were randomly assigned to the two systems, a causal conclusion is available; if the two groups were merely observed, the interval establishes a difference between the groups and not that the system produced it.
§5
Skill Check.
▸
Ten scenarios. Pick the chips that match your answer, then check. A scenario marks complete the first time every part is right. Progress saves on this device.