Mistake Master
Two Ways to Be Wrong, One Cutoff Between Them, and Only More Data Buys Both
You'll learnthe two errors a significance test can make and how to name each in context, why α is the false-alarm rate and β the miss rate against a stated alternative, how sliding the cutoff trades one for the other, and why sample size is the only lever that lowers both.
A test decides from a sample, so it can be wrong in two different ways: raise an alarm when nothing is happening, or miss something that is. Both errors are areas you can see. Draw the null's sampling distribution and the distribution the world would produce if the truth were 0.35, mark the rejection cutoff — computed here at 0.349, the line the last topic's two samples straddled — and α is the null curve's area past the cut while β is the truth curve's area on the near side. Slide the cut and the two areas trade: an α of 0.01 buys a β of 0.77. What escapes the trade is the sample size. At n = 250 this test misses a true 0.35 more than half the time; at n = 1000, five percent of the time, at the same α. And power is not 1 − α: it is 0.4873 here, on a different curve entirely. Twelve short steps; press Next for each one.