Least squares minimizes squared vertical residuals

The least-squares regression line is the line that minimizes

$$\sum (y - \hat{y})^2,$$

the sum of the squared residuals. Three parts of that sentence are load-bearing:

1. Squared, not absolute value. Squaring makes large misses count disproportionately, which is why a single far-out point can pull the line toward itself.
2. Vertical, not perpendicular. The residuals are prediction errors in the response, so the distance measured is straight up and down.
3. Minimized, not made zero. No line makes every residual 0 unless the points are already collinear.

For the eight students, the least-squares line is $\widehat{\text{score}} = 58.5 + 3.5(\text{hours})$, and its residuals $0, -1.5, 1, 1.5, -2, 0.5, 3, -2.5$ give $\sum (y - \hat{y})^2 = 25.0$. Any other line through this cloud produces a larger total; that is the entire content of the word "least".

$$b_1 = r\frac{s_y}{s_x}, \qquad b_0 = \bar{y} - b_1\bar{x}.$$

The first says the slope is the correlation rescaled by the two standard deviations, which is why $r$ and $b_1$ always share a sign and are otherwise different numbers. For the students, $r = 0.977$, $s_y = 8.78$, and $s_x = 2.449$, so $b_1 = 0.977 \times \frac{8.78}{2.449} = 3.5$ points per hour.

The second says the line passes through $(\bar{x}, \bar{y})$, the point of averages, always. Here $\bar{x} = 4.5$ and $\bar{y} = 74.25$, and indeed $58.5 + 3.5(4.5) = 74.25$. That is a fast check on any fitted line: substitute the mean of $x$ and see whether the mean of $y$ comes back.

The coefficient of determination $r^2$ is the proportion of the variability in the response that is accounted for by the linear model with the explanatory variable. For the students, $r^2 = 0.977^2 \approx 0.954$: about 95.4% of the variability in exam scores is accounted for by the linear relationship with study hours.

It can also be computed directly as $1 - \frac{\text{SSE}}{\text{SST}}$, and for these data $1 - \frac{25.0}{539.5} = 0.954$, which is the same number by a different road.

Four readings that are wrong:

1. "95.4% of the points lie on the line." It counts variability, not points, and here none of the eight lies exactly on it except the first.
2. "95.4% of the time the prediction is correct." It is not a success rate.
3. "The model is 95.4% accurate." Accuracy is not what it measures, and a curved relationship can produce a high $r^2$ with a systematically wrong model.
4. "Study hours cause 95.4% of the variation in scores." Cause is a design question, and this is observational data.

The interpretation to write, every time, names the percentage, the response's variability, the model, and the explanatory variable.

Because the criterion squares the residuals, points far from the rest carry outsized weight. Two kinds matter:

1. Outliers in the response: points with large residuals. They inflate SSE and lower $r^2$, and they pull the line vertically.
2. High-leverage points: points with extreme $x$-values, far to the left or right. These can swing the slope substantially even with a modest residual, because the line pivots around the point of averages.

A point that is both, extreme in $x$ and off the pattern, is the most influential of all: adding one such observation can reverse the sign of a slope. The check is to fit the line with and without the point and compare; a large change in the slope or intercept means the point is influential, and the analysis reports both fits rather than quietly dropping the observation.

None of this is repaired by a large $r^2$. A high $r^2$ driven by one leverage point describes the model's fit to that point more than to the rest of the data, which is why the residual plot from Topic 5.4 and the scatterplot from Topic 5.1 both come before any of these numbers are trusted.