Two Ways to Be Wrong, One Cutoff Between Them, and Only More Data Buys Both

You'll learnthe two errors a significance test can make and how to name each in context, why α is the false-alarm rate and β the miss rate against a stated alternative, how sliding the cutoff trades one for the other, and why sample size is the only lever that lowers both.

A test decides from a sample, so it can be wrong in two different ways: raise an alarm when nothing is happening, or miss something that is. Both errors are areas you can see. Draw the null's sampling distribution and the distribution the world would produce if the truth were 0.35, mark the rejection cutoff — computed here at 0.349, the line the last topic's two samples straddled — and α is the null curve's area past the cut while β is the truth curve's area on the near side. Slide the cut and the two areas trade: α of 0.01 buys a β of 0.77. What escapes the trade is the sample size. At n = 250 this test misses a true 0.35 more than half the time; at n = 1000, five percent of the time, at the same α. The two-by-two table, the trade, and the one true way out of it are all on screen, with every number computed from the curves it describes.

8 STEPS · 6 QUICK CHECKS · TYPE I AND TYPE II · THE α–β TRADE · POWER · v1

α = null area past the cut · β = truth area before it · they trade

Before you start

What you're looking at

Two sampling distributions for the renewal test: what samples look like if the null's 0.40 is true (blue), and what they look like if the truth is actually 0.35 (amber), with the rejection cutoff drawn between them.

The question

The test can raise a false alarm or miss a real effect. What is the probability of each, what makes them trade, and what lowers both?

Watch for

The two shaded areas. Every rate in this topic — α, β, power — is one of them, read off a curve, and the cutoff line is the only thing a tester directly controls.

Step 1 / 8